gms | German Medical Science

Question: Although randomized trial designs are considered to be the golden standard for experimental studies, its role in medical education research is still controversial. Drawbacks in the quality of randomized trials limit the impact on a best evidence medical education. The aim of our study is to provide a survey of the quality of published randomized medical education trials. Criteria analyzed are:

1.

reporting quality,

2.

methodological rigour

3.

internal validity.

Method: Six recent volumes (2002-2007) of four leading journals (Medical Education, Medical Teacher, Academic Medicine, Teaching and Learning in Medicine) were surveyed. A total of 70 randomized trial reports were assessed using 37 descriptors of reporting quality, methodological rigour and internal validity. Descriptive and exploratory methods were applied, including significance testing (Chi-square, Mann-Whitney- U, Fishers exact test, α=.05).

Results:

1.

Less than 10% of the publications included randomized medical education trial reports; there was no significant increase in the period 2002-2007. The 70 selected papers covered many different skills and educational interventions.

2.

87% were monocenter trials, most using two-arm parallel designs.

3.

45 publications reported significant results, but without an association between significance and sample size (p=.75).

4.

The assessment of both the internal validity and the methodological rigour was often hampered by missing information.

5.

Less than 25% of the studies included a description of the randomization procedure or the concealment of allocation.

6.

Blinding was only reported in 26 articles.

7.

In 30 trials, no information about an intention-to treat-analysis was given.

8.

Complete follow-up was reported only in 11 articles.

9.

53 papers presented no description of the drop-outs; 37 did not inform about handling of missing values, 25 publications completely ignored this issue.

10.

A priori sample size estimations were made in only 17 reports, post hoc power analysis in two studies.

11.

Effect sizes were reported in 30 studies.

12.

A description of the primary endpoint was found in 48 publications. Multiple endpoints were analyzed in 56 trials, but only few addressed the alpha inflation problem.

In total, an increase over time of the reporting quality could not be observed.

Conclusion: Many publications in medical education, even in top journals, still have a poor reporting quality of published randomized trials. Thus, the assessment of the internal validity was difficult. The survey identifies a number of substantial threads to validity, however, which may affect the acceptance of randomized trials in a best evidence medical education framework. Improvements of the methodological standards and of the reporting quality thus are major requirements for improving the professional quality of medical education research.