MedicineLakex

 

Wiki.usask.ca

Association Between Licensure Examination Scores
and Practice in Primary Care
Robyn Tamblyn; Michal Abrahamowicz; W. Dale Dauphinee; et al. Online article and related content current as of May 28, 2010. JAMA. 2002;288(23):3019-3026 (doi:10.1001/jama.288.23.3019) Topic collections Primary Care/ Family Medicine; Quality of Care; Quality of Care, Other ORIGINAL CONTRIBUTION Association Between Licensure Examination
Scores and Practice in Primary Care
Robyn Tamblyn, PhD
Context Standards for licensure are designed to provide assurance to the public of a
Michal Abrahamowicz, PhD physician's competence to practice. However, there has been little assessment of therelationship between examination scores and subsequent practice performance.
W. Dale Dauphinee, MD Objective To determine if there is a sustained relationship between certification ex-
James A. Hanley, PhD amination scores and practice performance and if licensing examinations taken at the John Norcini, PhD end of medical school are predictive of future practice in primary care.
Nadyne Girard, MSc Design, Setting, and Participants A total of 912 family physicians, who passed
the Que´bec family medicine certification examination (QLEX) between 1990 and 1993
Paul Grand'Maison, MD and entered practice. Linked databases were used to assess physicians' practice per- Carlos Brailovsky, MD, PhD formance for 3.4 million patients in the universal health care system in Que´bec, Canada.
Patients were seen during the follow-up period for the first 4 years (1993 cohort of physicians) to 7 years (1990 cohort of physicians) of practice from July 1 of the cer- HE MEDICAL PROFESSION EN- tification examination to December 31, 1996.
sures the basic competence ofphysicians by requiring them to Main Outcome Measures Mammography screening rate, continuity of care in-
dex, disease-specific and symptom-relief prescribing rate, contraindicated prescribing
pass licensing and certifying ex- rate, and consultation rate.
aminations.1 Although it is generally as-sumed that these examinations predict Results Physicians achieving higher scores on both examinations had higher rates (rate
increase per SD increase in score per 1000 persons per year) of mammography screening
how physicians will practice in the fu- (␤ for QLEX, 16.8 [95% confidence interval {CI}, 8.7-24.9]; ␤ for Medical Council of Canada ture,2 the data in support of this assump- Qualifying Examination [MCCQE], 17.4 [95% CI, 10.6-24.1]) and consultation (␤ for tion are mostly indirect. Physicians who QLEX, 4.9 [95% CI, 2.1-7.8]; ␤ for MCCQE, 2.9 [95% CI, 0.4-5.4]). Higher subscores in have more training in a discipline are diagnosis were predictive of higher rates in the difference between disease-specific and more knowledgeable3,4 and achieve symptom-relief prescribing (␤ for QLEX, 3.9 [95% CI, 0.9-7.0]; ␤ for MCCQE, 3.8 [95% higher scores in their respective disci- CI, 0.3-7.3]). Higher scores of drug knowledge were predictive of a lower rate (relative pline on recertification examinations.5 risk per SD increase in score) of contraindicated prescribing for MCCQE (relative risk, 0.88; More knowledgeable physicians are more 95% CI, 0.77-1.00). Relationships between examination scores and practice perfor- likely to adhere to evidence-based guide- mance were sustained through the first 4 to 7 years in practice.
lines in the delivery of care6,7 and achieve Conclusion Scores achieved on certification examinations and licensure examina-
better patient outcomes.6 Certification tions taken at the end of medical school show a sustained relationship, over 4 to 7 status, which represents pass/fail status years, with indices of preventive care and acute and chronic disease management inprimary care practice.
on certification examinations, is an im-portant predictor of quality of care.8,9 It is unknown, however, whether optimal and actual practice in the de- We previously reported that physi- scores achieved by physicians with the livery of preventive care,10 in the man- cians who achieved higher scores on the same training and specialty are predic- agement of acute and chronic dis- Que´bec family medicine certification ex- tive of future performance. A prior ease,7,11,12 and in the quality of drug amination were more likely to refer study8 found that scores on an inter- treatment.13 All of these are deficien- women for mammography screening, to nal medicine certification examina- cies that could potentially be pre- prescribe more disease-specific medica- tion predicted colleagues' ratings of the dicted by licensing examinations.
tion and fewer symptomatic and contra- quality of care delivered by internists5 to 8 years later. However, little is Author Affiliations: Departments of Medicine (Dr
University of Sherbrooke, Sherbrooke, Que´bec (Dr known about the relationship be- Tamblyn) and Epidemiology and Biostatistics (Drs Grand'Maison); and Centre d'e´valuation des sciences Tamblyn, Abrahamowicz, and Hanley, and Ms de la sante´, University of Laval, Ste-Foy, Que´bec (Dr tween examination scores and more ob- Girard), McGill University, Montreal, Que´bec; Medi- jective measures of quality of care. As- cal Council of Canada, Ottawa, Ontario (Dr Dau- Corresponding Author and Reprints: Robyn Tam-
sessing this relationship is relevant phinee); Foundation for Advancement of Interna- blyn, PhD, McGill University, Morrice House, 1140 Pine tional Medical Education and Research, Philadelphia, Ave W, Montre´al, Que´bec, Canada H3A 1A3 (e-mail: because important gaps exist between Pa (Dr Norcini); Department of Family Medicine, 2002 American Medical Association. All rights reserved.
(Reprinted) JAMA, December 18, 2002—Vol 288, No. 23 3019
EXAMINATION SCORES AND PRACTICE PERFORMANCE indicated drugs, and to refer more of their trieved from the RAMQ to assess an- postgraduate training. It is generally taken patients for consultation.14 However, as- nual practice activity and case-mix during the final year of medical school.
sessment of these outcomes was lim- differences between physician prac- A passing score on the MCCQE is ited to the first 18 months of practice. We tices. For each cohort, the follow-up pe- required for unrestricted licensure in all used this opportunity to determine if the riod was between July 1 of the certifica- Canadian provinces except Que´bec, association between family medicine cer- tion examination and December 31, although most Que´bec graduates take the tification examination scores and prac- 1996, providing practice assessment for examination. The 1988-1991 examina- tice performance persisted with increas- 4 years (1993 cohort) to 7 years (1990 tions consisted of 450 multiple-choice ing practice experience.14 We could not cohort). The study protocol was ap- questions, which tested knowledge in find any studies that explored the rela- proved by the institutional review board medicine, surgery, obstetrics/gynecol- tionship between earlier licensing ex- at McGill University, the Provincial Ac- ogy, psychiatry, pediatrics, and preven- aminations taken at the end of medical cess to Information Office, and the tive medicine, and also included 28 to school and future practice perfor- RAMQ legal counsel.
35 patient management problems that mance. We investigated whether such tested competence in clinical problem scores predict clinical behaviors 4 to 7 Family Medicine Certification
solving.17 To enable comparisons years later.
between the MCCQE and the Que´bec The Que´bec Licensing Examination certification examination, the 1258 mul- (QLEX) comprises the College of Fam- tiple-choice questions used between 1988 ily Physicians of Canada Certification and 1991 were reclassified by the test All Canadian provinces provide a uni- Examination (CFPCex) and the Que´- committees as diagnosis, management, versal health insurance program that bec Objective Structured Clinical Exami- or other knowledge. Test committees also covers the costs of medical care for pro- nation (OSCE).16 Between 1990 and identified management items that tested vincial residents. In Que´bec, 14500 phy- 1993, the CFPCex assessed diagnosis, drug-specific knowledge. Standardized sicians provide services to 7.4 million management, and prevention with ability scores were created for each sub- residents of the province, for whom 92% multiple-choice questions and short- score using the same reference group of services and 93% of physicians are paid answer management problems, and approach as the certification examina- by the Que´bec health insurance agency assessed communication skills with a tion.18 The overall score reliability was (Re´gie de l⬘assurance maladie du Que´- simulated patient. The OSCE mea- 0.92, and subscore reliabilities were 0.71 bec; RAMQ) on a fee-for-service basis.15 sured clinical problem-solving skills by for prevention, 0.63 for diagnosis, 0.73 direct observation of performance in 25 for management, and 0.48 for drug Design and Study Population
standardized patient encounters, rated knowledge. The Medical Council of A cohort of all family physicians who by physician examiners.16 To pass the Canada linked score data to the Que´bec passed the Que´bec family medicine cer- examination, a score of 60% was required cohort file by name, sex, and birth date.
tification examination between 1990 and in diagnosis, management, and commu- 1993, and entered fee-for-service prac- nication; 50% in prevention; and an tice in Que´bec, was followed up for the OSCE score greater than 2 SDs below the Data Sources and Retrieval. Four pre-
first 4 to 7 years of practice. Annual mea- mean. The overall reliability of exami- viously validated19 health administra- sures of each physician's practice per- nation scores varied from a low of 0.27 tive databases, linked by encrypted ben- formance were used to test associations for prevention to a high of 0.72 for the eficiary identifiers, were used to assess between examination scores and prac- OSCE.14 Scores were standardized to practice performance.20,21 The regis- tice performance. Salaried physicians adjust for differences in the difficulty of trant database provided patients' age, sex, were excluded because there was no ac- examinations using the reference group postal code, and date of death. The medi- curate way to identify all their patients, approach.14 The College of Physicians cal services database provided type, lo- as were physicians who subsequently provided the license number, medical cation, diagnosis, treating and referring trained in another specialty. Potentially school, and encrypted examination physician, and date of all services deliv- eligible physicians were identified by the scores to the RAMQ, who then retrieved ered on a fee-for-service basis. The pre- Que´bec College of Physicians, and phy- data on practice activity and linked this scription database provided drug, dose, sicians' license numbers were provided with score files through encrypted phy- duration, prescribing physician, and date to the RAMQ to retrieve data on prac- sician and patient identifiers.
of each prescription dispensed for pa- tice activity. Annual measures of each tients aged 65 years or older. The hos- physician's clinical behaviors were used Medical Council of Canada
pitalization database provided records of to test associations between examina- all hospital discharges including dis- tion scores and practice performance.
The Medical Council of Canada Quali- charge diagnoses and admission and dis- Data on all patients seen by each physi- fying Examination (MCCQE) is used to charge dates. The 1991 census data was cian, for each year of practice, were re- test an individual's competence to enter linked by 6-digit postal code to the reg- 3020 JAMA, December 18, 2002—Vol 288, No. 23 (Reprinted)
2002 American Medical Association. All rights reserved.
EXAMINATION SCORES AND PRACTICE PERFORMANCE istrant database to measure mean fam- tice who were referred for a bilateral thyroid medication). Symptom-relief ily income and educational achieve- mammogram by the physician. Eligible medication was defined as drugs that re- ment in the residential area of each women were between age 50 and 69 lieve symptoms, but have little impact on patient.22 For each physician, the medi- years, had no prior diagnosis of breast the disease process (eg, nonsteroidal anti- cal services claims files were used to iden- cancer, breast disease, or diagnostic inflammatory medications, benzodiaz- tify all patients seen by the physician mammogram, were due for screening, epines, low-dose narcotic analgesics) us- from the licensure date to December 31, and were not receiving primary care from ing the McGavock classification.30 1996. For each patient, the RAMQ re- a gynecologist or obstetrician.
Annual contraindicated prescribing rate trieved demographic data, as well as all Annual continuity of care was defined was the proportion of ambulatory elderly medical services, prescriptions, and hos- as the mean proportion of visits that were patients for whom the study physician pitalizations provided during the fol- made to or referred by the study physi- prescribed a relatively contraindicated low-up period. We also obtained data for cian by all patients in the primary care medication. These were defined by an the year prior to the first contact with the practice population. Using the medical updated expert review35,36 as 30 drugs that study physician to provide an indepen- services claims, we determined the pro- should be avoided in elderly patients dent assessment of patient characteris- portion of all visits in the year for each because of possible toxic effects.
tics that could not have been influ- patient that were made either to the Annual consultation rate was the pro- enced by the study physician.14 study physician or to other physicians portion of all ambulatory patients in the Indicators of Practice Performance.
based on study physician referral. Each respective calendar year referred, at least We assessed 6 performance indicators in patient's proportion was weighted by the once, to a specialist by the respective each follow-up year, selected on the basis square root of the total number of vis- primary care physician, based on medi- of unexplained practice variation, and/or its made by the patient in the year. An cal services claims. To be reimbursed their association with the outcomes or overall mean annual continuity of care for a consultation visit, the consultant costs of care.14 First, a mammography index for each physician's practice was must record a valid license number for screening rate was used to assess pre- calculated as the weighted mean of in- the referring physician.
ventive care, because physicians who dividual patient's proportions.
Time in Practice. Physicians' pre-
screen for breast cancer are also more Mammography screening and con- ceding practice experience was repre- likely to perform other preventive ser- tinuity of care were assessed only in the sented as a time-dependent covariate, vices.23 Second, continuity of care was primary care practice population, which based on a count of the number of pre- selected because of its importance in pre- consisted of patients seen in an office vious months that the physician had at vention and chronic disease manage- or clinic in the respective year, for least 1 fee-for-service billing in the ment,24-27 and to test the hypothesis that whom the study physician had pro- medical services claims file.
communication and management skills vided an annual physical or major (3 are predictive of better continuity.28,29 system) assessment. Consultation was Third, the differences between disease- assessed in all ambulatory patients for Relevant data were retrieved for each eli- specific and symptom-relief prescribing whom the study physician had billed gible patient for the 12-month period rate and contraindicated prescribing rate at least 1 outpatient, office practice, or preceding the first contact with the study were used as indicators of the quality of emergency department visit. Prescrib- physician. Individual characteristics of acute and chronic disease management ing rates were assessed in ambulatory all patients seen in a given year by the because variation in disease-specific rela- patients aged 65 years or older for physician were then aggregated to ad- tive to symptom-relief prescribing rate30 whom complete information was avail- just for between-physician differences in has been linked to diagnostic compe- able on all prescriptions dispensed.
practice population case-mix. Each prac- tence.14 Fourth, contraindicated pre- Annual difference between disease- tice population was characterized by age scribing, which accounts for 20% of drug- specific and symptom-relief prescribing and sex distribution, mean family in- related adverse events, may be caused by rates was determined by examining come, educational achievement, geo- deficiencies in physician knowledge.31 medications prescribed by the study phy- graphic access to health care,14 propen- Finally, consultation rate was used as an sician to all elderly patients. Disease- sity for the use of health care services,37 indicator of resource use because refer- specific medication was defined as drugs comorbidity (Charlson index),38 and hos- ral determines access to higher cost that would rarely be prescribed with- pitalization rates in the previous year.
specialty care,32 and primary care phy- out an investigation-confirmed disease These annual case-mix measures were sicians with higher self-reported com- (eg, anticoagulants, anticonvulsants, an- modeled as time-dependent covariates.
petence appear more likely to refer tidepressants, antihypertensives, medi- For contraindicated prescribing, no ad- patients for specialty consultation.33,34 cations for cardiovascular disease, justments were made for case-mix be- Annual mammography screening rate asthma, and Parkinson disease, cortico- cause the prescription of relatively con- was defined as the proportion of eli- steroids, diuretics, antiglaucoma medi- traindicated medication is rarely justified gible women in the primary care prac- cation, hypoglycemic medication, and by patient characteristics.14 2002 American Medical Association. All rights reserved.
(Reprinted) JAMA, December 18, 2002—Vol 288, No. 23 3021
EXAMINATION SCORES AND PRACTICE PERFORMANCE number of patients in the practice popu- diminished over time, the interactions Relationships between examination lation in a given year was used as a weight between examination score and prac- scores and practice performance were in the analysis. Linearity assumptions tice experience were tested. For signifi- tested using multiple linear and Pois- were evaluated by testing the statistical cant relationships between examina- son regression for repeated measures significance of the quadratic compo- tion score and outcomes that persisted using generalized estimating equa- nent. Poisson regression was used to over time, the cumulative difference in tions.39 Physicians were the unit of analy- assess rates of contraindicated prescrib- the number of outcomes per 1000 sis. Outcome variables were the 5 annual ing because such events were rare. The patients followed up over the first 5 years measures of practice performance. An SEs were empirically estimated to account in practice by high-scoring (2 SDs above autoregressive first-order correlation for overdispersion. Annual measures of the mean) rather than low-scoring (2 SDs structure of residuals was used to char- case-mix were included in all analyses, below the mean) physicians was esti- acterize the interdependence between as were indicators of the medical school mated. Regression coefficients for the annual performance measures for each to conservatively assess the impact of score-outcome relationship were used to physician. To reduce imprecision in out- variation in examination scores within determine the expected annual differ- come measurement, physicians were each medical school. To determine ence in rates corresponding to a 4-SD dif- excluded in years in which they had fewer whether the association between exami- ference in examination scores, and then than 5 patients, and the logarithm of the nation scores and practice outcomes the result was multiplied by 5 to esti-mate the cumulative impact over 5 years.
P⬍.05 was used as the level of statistical Table 1. Physician Characteristics and Examination Scores for 912 Que´bec Family Physicians*
significance. We used SAS statistical soft- ware to perform our analyses (Version 8.0, SAS Institute Inc, Cary, NC).
Between 1990 and 1993, a total of 944 family physicians passed the QLEX, 920 (97.5%) started practice in Que´bec, 912 (96.7%) entered a fee-for-service prac- Other part of Canada or in United States tice, of whom 58.1% were female Other international (TABLE 1). Overall, 85.8% of physi-
Postgraduate training same as undergraduate medical school cians took the MCCQE during the final Certification year year of medical school. Que´bec medical school graduates had modestly lower MCCQE scores in comparison with other Canadian graduates (mean [SD], −0.04 [0.84] vs 0.09 [0.92]; P⬍.001). Gradu- Took MCCQE examination ates who entered family medicine or gen- Mean (SD) [Range]
eral practice training in Canada had slightly lower standardized scores than −0.03 (1.05) [−3.97 to 2.76] those entering specialty programs (mean 0.01 (1.07) [−3.97 to 4.40] [SD], 0.02 [0.95] vs 0.13 [0.84]; −0.04 (1.03) [−4.53 to 2.47] P⬍.001). Que´bec family physicians who −0.02 (1.06) [−4.63 to 3.04] did not take the MCCQE had lower cer- Clinical assessment −0.01 (0.96) [−2.81 to 2.91] tification examination scores than those who did (mean [SD], −0.76 [1.18] vs −0.22 (0.89) [−4.31 to 2.24] −0.03 [1.02]; P⬍.001). Mean scores achieved on both the QLEX and MC- 0.02 (0.92) [−3.64 to 2.55] CQE were equivalent or slightly lower 0.21 (0.89) [−3.10 to 2.67] than in the reference group of first-time −0.19 (0.86) [−3.66 to 1.95] takers, with a typical range of 6 to 7 SDs −0.04 (0.92) [−3.04 to 2.93] (Table 1). The Pearson correlation be- Clinical assessment −0.83 (1.20) [−5.97 to 2.03] tween the MCCQE and QLEX was 0.55, *Examination scores are standardized to a mean of zero representing the average score for first-time takers of the examination from North American medical schools. MCCQE indicates Medical Council of Canada Qualifying Exami- and ranged from 0.26 (clinical problem nation; QLEX, Que´bec Certification Examination.
solving) to 0.49 (management).
3022 JAMA, December 18, 2002—Vol 288, No. 23 (Reprinted)
2002 American Medical Association. All rights reserved.
EXAMINATION SCORES AND PRACTICE PERFORMANCE Physicians practiced in a mean of 2.6 comes. Higher scores were associated and 17.4 per 1000 for the overall QLEX (year 1-2) to 3.2 (year 5-7) different with higher rates of disease-specific rela- score (Table 3). The diagnosis sub- types of settings (TABLE 2). During the
tive to symptom-relief prescribing, and score was the strongest predictor of dif- first 4 to 7 years of practice, an increas- a lower risk of contraindicated pre- ferences in the rates of disease-specific ing proportion of physicians moved scribing, although the latter did not and symptom-relief prescribing in both their primary practice base from rural achieve statistical significance (Table 3).
the MCCQE and the QLEX. The drug to urban populations, and established The strength and significance of the as- knowledge subscore was the only sig- private office practice. After the first 2 sociations between examination scores nificant predictor of contraindicated pre- years, practice size, number of days and practice outcomes increased when scribing—reducing the risk of contra- worked per year, and visits per day con- medical school was excluded from the indicated prescribing by 12% per SD tinued to increase, but at a slower rate.
model, because there were systematic increase in score. During the first 5 years, The 912 physicians billed for 3.4 mil- differences in scores for graduates from a high-scoring physician would be ex- lion different patients (45.9% of the Que´- different medical schools.
pected to write 85 fewer contraindi- bec population), of whom 1.4 million cated prescriptions per 1000 elderly pa- were in their primary care practice popu- tients than a low-scoring physician.
lations, 385 321 were elderly, and Scores on the MCCQE, taken at the end 119866 were women eligible for mam- of medical school, had a similar pat- mography screening.
tern of relationships as the certification We examined the relationship be- The relationship between the QLEX examination (TABLE 4). For example, the
tween licensure and certification ex- and mammography screening was sus- increase in mammography screening rate amination scores and practice perfor- tained over the first 4 to 7 years in prac- per SD increase in score was 16.8 per mance in a sample of newly certified tice (TABLE 3). The significant interac-
1000 for the MCCQE score (Table 4) family physicians. Linked databases tion between certification examinationscore and practice experience indi-cated that the strength of the relation- Table 2. Practice Setting and Workload Characteristics for Family Physicians in the First 4 to
ship increased over time. For every SD 7 Years of Practice increase in QLEX score, the mammog- Physician Practice Year
raphy screening rate increased by 13.3 women per 1000 in years 1 to 2 of prac- tice, 21.7 by years 3 to 4, and 19.2 by years 5 to 7. The persistence of this re- Practice setting* lationship means that, during the first Outpatient clinic 5 years of practice, high-scoring physi- Emergency department cians would be expected to order 347 more mammograms per 1000 women Intensive care unit than low-scoring physicians. In con- trast, there was no relationship be- Private office practice tween QLEX scores and continuity of care. Overall, however, family physi- Mean (SD)
cians provided or coordinated only Patient geographic distribution, %† 28.3% of all visits made by patients in their primary care practice population.
Consultation rate showed a persis- tent, but modest, association with QLEX Practice settings per year score over the first 7 years of practice Patients seen per year (Table 3). Each SD increase in score was Work days per year associated with an additional 2.92 re- Visits per work day ferrals for specialty consultation per *Most physicians practiced in multiple practice settings (eg, in year 1-2, the average number of different practice set- 1000 patients seen per year, resulting, tings was 2.6 per physician). For this reason, the number (percentage) of physicians practicing in each type of prac- over the first 5 years, in 58 more refer- tice setting adds to more than 100%, as most physicians were represented in more than 1 category.
†The residence of each patient in a physician's practice population was categorized as urban (resided in the regions of rals per 1000 patients by high-scoring Montre´al, Que´bec, Laval, or Monte´re´gie), intermediate (resided in the regions of Lanaudières, Estrie Saguenay-Lac-St-Jean, Laurentides, Mauricie-Bois-Francs, or Outaouais), or rural-remote (resided in the regions of Chaudières- than low-scoring physicians.
Appalaches, Abitibi-Te´miscamingue, Gaspe´sie, Bas-Saint-Laurent, Côte-Nord, Nord-du-Que´bec, Kativik Terres- Diagnosis and management sub- cries-de la Baie-James). For each physician, the proportion of patients from urban, intermediate, and rural-remoteregions was determined for the first 1 to 2, 3 to 4, and 5 to 7 years of practice. The mean represents the average scores of the QLEX were the only sig- proportion of patients in the practices of physicians in the cohort who resided in urban, intermediate, and rural-remote locations.
nificant predictors of prescribing out- 2002 American Medical Association. All rights reserved.
(Reprinted) JAMA, December 18, 2002—Vol 288, No. 23 3023
EXAMINATION SCORES AND PRACTICE PERFORMANCE from a universal health care system al- through the first 4 to 7 years of prac- that this introduced substantial bias in lowed us to follow-up all licensed phy- tice, and that examinations taken in the the results. These results have several sicians, and all their patients in their final year of medical school were also implications for licensing bodies.
first 4 to 7 years of practice. Thus, we significant predictors of practice per- Such outcome data could be incor- minimized selection biases that could formance. However, the use of admin- porated into the establishment of ex- confound the assessment of relation- istrative databases had limitations. The amination passing standards so that ships. We were also able to assess the databases restricted the type of perfor- standard-setting participants can population impact of potential licen- mance indicators that could be mea- weigh their usual judgments of test con- sure regulations. The most important sured and we were limited in our abil- tent and pass rates against the conse- findings of this study were that the re- ity to adjust for confounding by case- quences for patients.40-42 For example, lationships between certification ex- mix, but as case-mix was not associated an increase in the passing criterion of amination scores were sustained with examination scores, it is unlikely only 1 SD in drug knowledge would Table 3. Association Between Family Medicine Certification Examination Scores and Practice Performance in the First 4 to 7 Years of Practice
Change in Outcome per SD Increase in Score
All Practice Years
Practice Year †
Mean (SD)
Rate per 1000
Type of Certification
P Value for
␤ (95% CI)*
Mammography screening rate 17.37 (10.6 to 24.1) ⬍.001 13.3 21.7 19.2 among eligible women 15.61 (8.9 to 22.3) ⬍.001 17.9 15.9 12.7 Clinical assessment 14.22 (6.9 to 22.6) Coordination of care Continuity of care, % of visits 0.2 (−0.3 to 0.6) 0.05 (−0.4 to 0.5) 0.2 (−0.3 to 0.6) Consultation rate 2.92 (0.4 to 5.4) Acute and chronic disease management Symptom-relief prescription rate −2.83 (−8.7 to 3.0) among elderly patients −2.56 (−7.9 to 2.8) −5.97 (−12.0 to 0) Disease-specific prescription rate minus 2.89 (−0.6 to 6.4) symptom-relief prescription rate 3.94 (0.9 to 7.0) 2.42 (−0.9 to 5.7) Contraindicated prescriptions among 0.97 (0.9 to 1.1) elderly patients§ 0.99 (0.9 to 1.1) 0.94 (0.8 to 1.0) *The regression coefficient ␤ represents the estimated change in the rate or value of the practice outcome per SD increase in score in the first 4 to 7 years in practice with a 95% confidence interval (CI). In these overall models, the interaction term to test the potential modification of the magnitude of the effect between practice outcome and examinationscores in relationship with the number of months in practice is not included. In instances in which there was a significant interaction between examination score and months inpractice (ie, mammography screening rate), the estimates for each category of years in practice provide a more appropriate estimate of the effects. Each ␤ was estimated by amultivariate regression model within a generalized estimating equation framework, in which physician was the unit of analysis and annual assessment of outcome rates/values,were represented as repeated measurements for each physician. Observations were weighted by the logarithm of each physician's annual practice size. The estimate of theexamination score, practice outcome relationship was adjusted for differences in annual practice case-mix including age and sex structure, socioeconomic status, geographicaccess to health care, comorbidity, and propensity to use health care services based on data for individual practice patients in the year prior to outcome assessment. The onlyexception was for contraindicated prescribing in which practice size was used to weight estimate regression coefficients and medical school was included, but practice case-mixcovariates were not included as these attributes of the practice population would rarely justify contraindicated prescribing in the elderly. When medical school was excluded fromthe regression models, the magnitude of the association between examination scores and practice outcomes increased because some medical schools had systematically lowerscores than others. If the analysis were based on usual practice, in which pass-fail decisions are made irrespective of medical school, the overall predictive relationship betweenexamination scores and outcomes would have been higher: mammography screening and overall score (␤, 19.3; 95% CI, 13.1-25.4), consultation rate and overall score (␤, 3.70;95% CI, 1.4-5.9), symptom relief prescribing and management score (␤, −7.30; 95% CI, −13.4 to −1.2), disease-specific minus symptom-relief prescribing rate and diagnosisscore (␤, 4.07; 95% CI, 1.2-7.0), and contraindicated prescribing and management score (relative risk, 0.89; 95% CI, 0.8-1.0).
†To facilitate interpretation of changes in the magnitude of the association between examination scores and practice outcomes over the first 4 to 7 years of practice, examination score outcome relationships were estimated for 3 time intervals of practice based on a categorization, for each physician, of the cumulative months in practice from the practiceentry month. The interaction effects presented by intervals of years in practice were produced by a separate analysis to facilitate easier interpretation, in which the interactionsbetween examination score and 2 dummy variables, representing practice years 3 to 4 and 5 to 7 relative to years 1 to 2 were estimated.
‡To test the hypothesis that the relationship between certification examination scores and practice outcomes would be attenuated with increasing time in practice, we tested the interaction between examination score and cumulative months in practice. Cumulative practice months, treated as a time-dependent covariate, were determined by countingeach month that the physician billed the Que´bec health insurance agency (Re´gie de I'assurance maladie due Que´bec; RAMQ) for fee-for-service or salaried care for Que´becmedical care beneficiaries. P values are reported for each of the interaction terms (examination score multiplied by cumulative months in practice) that were estimated for eachcombination of outcome and examination score.
§Included phenylbutazone, dipyridamole, reserpine, disopyramide, clofibrate, methylphenidate, chlordiazepoxide, diazepam, clorazepate, flurazepam, clonazepam, clobazam, primi- done, fluoxetine, phenelzine, tranylcypromine, moclobemide, amitriptyline, doxepin, imipramine, trimipramine, clomipramine, amoxapine, maprotiline, cyclobenzaprine, metho-carbamol, pentazocine, meperidine, triazolam, and theophylline. Data expressed as relative risk of contraindicated prescribing per 1 SD increase in score.
3024 JAMA, December 18, 2002—Vol 288, No. 23 (Reprinted)
2002 American Medical Association. All rights reserved.
EXAMINATION SCORES AND PRACTICE PERFORMANCE have resulted in failing 16 additional of this magnitude is equivalent to or as drug knowledge, diagnosis, and man- physicians on the MCCQE licensing ex- greater than that reported for the most amination over 4 years. It would also effective form of continuing medical Consultation rates increased lin- have reduced the expected risk of con- education for physicians with high rates early with examination score. Physi- traindicated prescriptions for elderly pa- of inappropriate prescribing.43 To mini- cians tend to report higher referral rates tients seen by these physicians by ap- mize the risk of adverse outcomes for in clinical areas in which they felt more proximately 42% (from 4.7% for these the population, passing standards may competent.33,34 More competent phy- low-scoring physicians to 3.3% for an need to be established for individual sicians may be more aware of their limi- average physician). A reduction in risk components of the examination, such tations. However, most examinations do Table 4. Association Between Medical Council of Canada Licensing Examination Scores and Practice Performance in the First 4 to 7 Years of
Practice
Change in Outcome per SD Increase in Score
All Practice Years
Practice Year †
Mean (SD)
Rate per 1000
Type of Certification
P Value for
␤ (95% CI)*
Mammography screening rate 16.81 (8.7 to 24.9) ⬍.001 15.2 21.5 15.3 8.10 (−1.0 to 17.2) Clinical assessment 11.54 (5.5 to 17.6) Coordination of care Continuity of care, % of visits 0.2 (−0.4 to 0.8) 0.01 (−0.6 to 0.6) Consultation rate 4.93 (2.1 to 7.8) Acute and chronic disease management Symptom-relief prescribing rate −7.15 (−15.0 to 1.0) among elderly patients −5.1 (−12.0 to 2.3) −7.0 (−14.0 to 2.2) −6.99 (−14.0 to 0.4) Disease-specific prescription rate minus 4.83 (0.9 to 8.8) symptom-relief prescription rate 3.80 (0.3 to 7.3) 3.0 (−1.0 to 6.9) Contraindicated prescribing among 0.93 (0.82 to 1.05) elderly patients§ 0.97 (0.86 to 1.10) 0.91 (0.80 to 1.03) 0.88 (0.77 to 1.0) *The regression coefficient ␤ represents the estimated change in the rate or value of the practice outcome per SD increase in score in the first 4 to 7 years in practice with a 95% confidence interval (CI). In these overall models, the interaction term to test the potential modification of the magnitude of the effect between practice outcome and examinationscores in relationship with the number of months in practice is not included. In instances in which there was a significant interaction between examination score and months inpractice (ie, mammography screening rate), the estimates for each category of years in practice provide a more appropriate estimate of the effects. Each ␤ was estimated by amultivariate regression model within a generalized estimating equation framework, in which physician was the unit of analysis and annual assessment of outcome rates/values,were represented as repeated measurements for each physician. Observations were weighted by the logarithm of each physician's annual practice size. The estimate of theexamination score, practice outcome relationship was adjusted for differences in annual practice case-mix including age and sex structure, socioeconomic status, geographicaccess to health care, comorbidity, and propensity to use health care services based on data for individual practice patients in the year prior to outcome assessment. The onlyexception was for contraindicated prescribing in which practice size was used to weight estimated regression coefficients and medical school was included, but practice case-mix covariates were not included as these attributes of the practice population would rarely justify contraindicated prescribing in the elderly. When medical school was excludedfrom the regression models, the magnitude of the association between examination scores and practice outcomes increased because some medical schools had systematicallylower scores than others. If the analysis were based on usual practice, in which pass-fail decisions are made irrespective of medical school, the overall predictive relationshipbetween examination scores and outcomes would have been higher: mammography screening and overall score (␤, 18.3; 95% CI, 10.3-26.3), consultation rate and overall score(␤, 5.13; 95% CI, 2.3-7.9), symptom relief prescribing and drug knowledge score (␤, −8.36; 95% CI, −15.8 to −0.9), disease-specific minus symptom-relief prescribing rate anddiagnosis score (␤, 3.39; 95% CI, −0.4 to 6.8), and contraindicated prescribing and drug knowledge score (relative risk, 0.85; 95% CI, 0.7-0.9).
†To facilitate interpretation of changes in the magnitude of the association between examination scores and practice outcomes over the first 4 to 7 years of practice, examination score outcome relationships were estimated for 3 time intervals of practice based on a categorization, for each physician, of the cumulative months in practice from the practiceentry month. The interaction effects presented by intervals of years in practice were produced by a separate analysis to facilitate easier interpretation, in which the interactionsbetween examination score and 2 dummy variables, representing practice years 3 to 4 and 5 to 7 relative to years 1 to 2 were estimated.
‡To test the hypothesis that the relationship between certification examination scores and practice outcomes would be attenuated with increasing time in practice, we tested the interaction between examination score and cumulative months in practice. Cumulative practice months, treated as a time-dependent covariate, were determined by countingeach month that the physician billed the Que´bec health insurance agency (Re´gie de I'assurance maladie due Que´bec; RAMQ) for fee-for-service or salaried care for Que´becmedical care beneficiaries. P values are reported for each of the interaction terms (examination score multiplied by cumulative months in practice) that were estimated for eachcombination of outcome and examination score.
§Included phenylbutazone, dipyridamole, reserpine, disopyramide, clofibrate, methylphenidate, chlordiazepoxide, diazepam, clorazepate, flurazepam, clonazepam, clobazam, primi- done, fluoxetine, phenelzine, tranylcypromine, moclobemide, amitriptyline, doxepin, imipramine, trimipramine, clomipramine, amoxapine, maprotiline, cyclobenzaprine, metho-carbamol, pentazocine, meperidine, triazolam, and theophylline. Data expressed as relative risk of contraindicated prescribing per 1 SD increase in score.
2002 American Medical Association. All rights reserved.
(Reprinted) JAMA, December 18, 2002—Vol 288, No. 23 3025
EXAMINATION SCORES AND PRACTICE PERFORMANCE not test whether an individual knows bec for his assistance in retrieving the health service 21. Wilchesky M, Tamblyn RM. Predictive values of
data and responding to our many questions; and M.
diagnoses in medical services claims. J Clin Pharma- when to refer. This may be an impor- Jimmy Fragos, BSc, for his expert assistance in data- tant area for test development, as the base management and variable creation.
22. Wilkins K. Use of postal codes and addresses in the
risk of outcomes related to medical er- analysis of health data. Health Rep. 1993;5:157-177.
23. Salive ME, Guralnik JM, Brock D. Preventive ser-
rors has been shown to increase when vices for breast and cervical cancer in US office- physicians practice beyond their areas 1. Irvine D. The performance of doctors, I: profes-
based practices. Prev Med. 1996;25:561-568.
sionalism and self-regulation in a changing world. BMJ. 24. Bindman AB, Grumbach K, Osmond D, et al. Pre-
ventable hospitalizations and access to health care.
The finding that examinations taken 2. Tamblyn RM. Is the public being protected? pre-
vention of suboptimal medical practice through train- 25. Shea S, Misra D, Ehrlich MH, Field L, Francis CK.
at the end of medical school were also ing programs and credentialing examinations. Eval Predisposing factors for severe, uncontrolled hyper- predictive of future practice may be use- Health Prof. 1994;17:198-221.
tension in an inner-city minority population. N Engl ful for residency program directors, who 3. Ayanian JZ, Hauptman PJ, Guadagnoli E, et al.
J Med. 1992;327:776-781.
Knowledge and practices of generalist and specialist 26. Wasson JH, Sauvigne AE, Mogielnicki RP, et al.
could use this information to target physicians regarding drug therapy for acute myocar- Continuity of outpatient medical care in elderly men: learning opportunities to areas of de- dial infarction. N Engl J Med. 1994;331:1136-1142.
a randomized trial. JAMA. 1984;252:2413-2417.
4. Edep ME, Shah NB, Tateo IM, Massie BM. Differ-
27. Lambrew JM, DeFriese GH, Carey TS, Ricketts TC,
ficiency. Medical school educators may ences between primary care physicians and cardiolo- Biddle AK. The effects of having a regular doctor on ac- also be able to identify persons more gists in management of congestive heart failure. J Am cess to primary care. Med Care. 1996;34:138-154.
Coll Cardiol. 1997;30:518-526.
28. Hulscher ME, van Drenth BB, Mokkink HG, et al.
likely to experience difficulty in prac- 5. Norcini J, Lipner RS, Benson JA Jr, Webster GD. An
Barriers to preventive care in general practice. Br J Gen tice earlier in training, as scores on analysis of the knowledge base of practicing inter- nists as measured by the 1980 recertification exami- 29. Safran DG, Montgomery JE, Chang H, Murphy
medical school examinations are nation. Ann Intern Med. 1985;102:385-389.
J, Rogers H. Switching doctors: predictors of volun- strongly correlated with scores on li- 6. Jollis JG, DeLong ER, Peterson ED, et al. Outcome
tary disenrollment for primary care physician's prac- censing examinations.45 of acute myocardial infarction according to the spe- tice. J Fam Pract. 2001;50:130-136.
cialty of the admitting physician. N Engl J Med. 1996; 30. McGavock H, Wilson-Davis K, Niblock RWF. Un-
Future research should investigate suspected patterns of drug utilization revealed by in- methods of establishing outcomes- 7. Weiner JP, Parente ST, Garnick DW, Fowles J,
terrogation of a regional general practitioner prescrib- Lawthers AG, Palmer HR. Variation in office-based qual- ing database. Pharmacoepidemiol Drug Saf. 1992;1: based passing scores, and develop more ity: a claims-based profile of care provided to Medicare refined measures of quality of care in patients with diabetes. JAMA. 1995;273:1503-1508.
31. Leape LL, Bates DW, Cullen DJ, et al. Systems analy-
8. Ramsey PG, Carline JD, Inui TS, Larson EB, Lo-
sis of adverse drug events. JAMA. 1995;274:35-43.
disease-specific populations. Longitu- Gerfo JP, Wenrich MD. Predictive validity of certifi- 32. Franks P, Clancy CM. Gatekeeping revisited: pro-
dinal follow-up studies should be con- cation by the American Board of Internal Medicine.
tecting patients from overtreatment. N Engl J Med. ducted in multijurisdictional cohorts of Ann Intern Med. 1989;110:719-726.
9. McAuley RG, Paul WM, Morrison GH, Beckett RF,
33. Reynolds GA, Chitnis JG, Roland MO. General prac-
medical graduates to explore the as- Goldsmith CH. Five-year results of the peer assess- titioner outpatient referrals: do good doctors refer more pects of admissions, undergraduate, and ment program of the College of Physicians and Sur- patients to hospital? BMJ. 1991;302:1250-1252.
geons of Ontario. CMAJ. 1990;143:1193-1199.
34. Calman NS, Hyman RB, Licht W. Variability in con-
postgraduate training that may influ- 10. Murata PJ, Li J. Relationship between Pap smear
sultation rates and practitioner level of diagnostic cer- ence practice.
performance and physician ordering a mammogram.
tainty. J Fam Pract. 1992;35:31-38.
J Fam Pract. 1992;35:644-648.
35. Beers MH, Ouslander JG, Rollingher I, Reuben DB,
11. Nicolucci A, Cavaliere D, Scorpiglione N, et al. A
Brooks J, Beck JC. Explicit criteria for determining in- Author Contributions: Study concept and design:
comprehensive assessment of the avoidability of long- appropriate medication use in nursing home resi- Tamblyn, Abrahamowicz, Dauphinee.
term complications of diabetes. Diabetes Care. 1996; dents. Arch Intern Med. 1991;151:1825-1832.
36. McLeod PJ, Huang AR, Tamblyn RM, Gayton DC.
12. Legorreta AP, Christian-Herman J, O'Connor RD,
Defining inappropriate practices in prescribing for el- Analysis and interpretation of data: Tamblyn, et al. Compliance with national asthma management derly people. CMAJ. 1997;156:385-391.
Abrahamowicz, Hanley, Norcini, Girard.
guidelines and specialty care. Arch Intern Med. 1998; 37. Starfield B, Weiner J, Mumford L, Steinwachs D.
Drafting of the manuscript: Tamblyn, Abrahamowicz, Ambulatory care groups: a categorization of diag- 13. Wilcox SM, Himmelstein DU, Woolhandler S. In-
noses for research and management. Health Serv Res. Critical revision of the manuscript for important in- appropriate drug prescribing for the community- tellectual content: Tamblyn, Abrahamowicz, dwelling elderly. JAMA. 1994;272:292-296.
38. Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical
Dauphinee, Hanley, Norcini, Girard, Grand'Maison, 14. Tamblyn R, Abrahamowicz M, Brailovsky C, et al.
comorbidity index for use with ICD-9-CM adminstrative Association between licensing examination scores and databases. J Clin Epidemiol. 1992;45:613-619.
Statistical expertise: Tamblyn, Abrahamowicz, Hanley, resource use and quality of care in primary care prac- 39. Zeger SL, Liang KY. Longitudinal data analysis for
Norcini, Girard.
tice. JAMA. 1998;280:989-996.
discrete and continous outcomes. Biometrics. 1986; Obtained funding: Tamblyn, Abrahamowicz, 15. Statistiques Annuelles. Que´bec, Que´bec: Regie
de l'assurance-maladie du Que´bec; 1995.
40. Public trust and accountability for clinical perfor-
Administrative, technical, or material support: 16. Grand'Maison P, Lescop J, Rainsberry P, Brai-
mance: lessons from the national press reportage of the Tamblyn, Dauphinee, Girard.
lovsky CA. Large-scale use of an objective, struc- Bristol hearing. J Eval Clin Pract. 1999;5:335-342.
Study supervision: Tamblyn.
tured clinical examination for licensing family physi- 41. Southgate L, Hays RB, Norcini J, et al. Setting per-
Funding/Support: Research support was provided by
cians. CMAJ. 1992;146:1735-1740.
formance standards for medical practice: a theoreti- the Canadian Institutes of Health Research, and the 17. Information Pamphlet on the Qualifying Exami-
cal framework. Med Educ. 2001;35:474-481.
Fonds de Recherche en Sante´ du Que´bec. In addi- nation. Ottawa, Ontario: Medical Council of Canada; 42. Norcini JJ Jr. Standards and reliability in evalua-
tion, Dr Tamblyn and Dr Abrahamowicz are medical tion. Acad Med. 1999;74:1088-1090.
scientists at the Canadian Institutes of Health Re- 18. Livingston SA, Zieky M. Passing Scores. Prince-
43. Schaffner W, Ray WA, Federspiel CF, Miller WO.
ton, NJ: Educational Testing Service; 1989.
Improving antibiotic prescribing in office practice.
Acknowledgment: We thank Tim Wood, PhD, and
19. Tamblyn RM, Lavoie G, Petrella L, Monette J. Use
Andre´ Phillipe Boulais, MSC, of the Medical Council of prescription claims databases in pharmacoepidemio- 44. Leape LL, Brennan TA, Laird N, et al. Nature of
of Canada for their expert assistance in data re- logical research. J Clin Epidemiol. 1995;48:999-1009.
adverse events in hospitalized patients. N Engl J Med. trieval, linkage, and score reclassification; Joelle Le- 20. Levy AR, Tamblyn RFD, McLeod P, Hanley J. Cod-
scop, MD, MPH, of the Que´bec College of Physi- ing accuracy of hospital discharge data for elderly sur- 45. Veloski JJ, Hojat M, Gonnella JS. The validity of
cians who made this study possible; M. Jacques Barry, vivors of myocardial infarction. Can J Cardiol. 1999; Part III of the National Board Examination. Proc Annu MBA, of the Re´gie de l'assurance maladie du Que´- Conf Res Med Educ. 1987;26:54-59.
3026 JAMA, December 18, 2002—Vol 288, No. 23 (Reprinted)
2002 American Medical Association. All rights reserved.

Source: https://wiki.usask.ca/download/attachments/3997711/LMCC+Pt+I_+QLEX+predicts+Tamblyn+et+al+JAMA+2002.pdf