Hospital Transfers Bias Quality Measures

SEATTLE—Standardized mortality ratios (SMRs), which the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) is considering using as a core measure of ICU quality in US hospitals, may be easily skewed by the transfer of critically ill patients, according to research from the University of Washington in Seattle.¹ Through data analysis and subsequent simulation, investigators found that the transfer of one additional patient per month would substantially bias the SMRs of many ICUs, and the authors proposed that an uncritical look at the measure “is likely to misinform rather than provide meaningful information about ICU quality.”

In the cohort study, Jeremy M. Kahn, MD, MS, and colleagues gathered information from the APACHE IV clinical database. After excluding patients younger than 18 or with repeat hospital admissions and ICUs with fewer than 150 patients per year, the investigators analyzed data on 120,475 patients from 85 ICUs. Overall, the median out-of-hospital ICU transfer rate was 3%, and the mean observed hospital mortality rate was 13.3%. The mean baseline SMR was 1.06 overall.

A Simulated Improvement

Using a Monte Carlo simulation, the investigators increased the frequency of out-of-hospital transfers by 2% and 6%, a mean of 1.2 and 3.5 additional transfers per month. The mean probability of death of transfer patients (0.12 at baseline) was raised to 0.25 and 0.21, respectively, with the transfer increases. However, the 2% increase lowered the SMRs of 27 hospitals by at least one tenth; with the 6% increase all but one of the hospitals experienced a similar effect. The degree of bias was found to be “marginally greater” in smaller ICUs, low-performing ICUs, and ICUs with higher transfer rates at baseline.

Further analysis showed that an individual ICU’s performance rank, relative to the others examined, could significantly improve due to the demonstrated transfer bias. As an example, the authors posed that if the ICU that ranked in the 11th percentile (original SMR of 1.27) were the only one to increase its number of transfers by 2%, its ranking would rise to the 28th percentile; with a 6% increase, it would jump to the 37th percentile.

More Effective Evaluation Methods

The investigators recommended that all the factors affecting SMRs be fully understood before the JCAHO adopts the SMR as a core measure. They also warned that pay-for-performance initiatives could be affected by “gaming”: “Just as in some settings providers have avoided treating high-risk patients or up-coded the severity of illness to improve risk-adjusted outcome measures,” they stated, “ICU providers could actively increase [long-term acute care] discharges to improve their SMR.” In an SMR-based pay-for-performance scheme, hospitals could be rewarded for changes in their discharge practices rather than for changes in quality.

Instead, Dr. Kahn and colleagues suggested that regulating bodies focus on and reward the use of process-based quality measures, such as rates of appropriate use of evidence-based therapies. Use of process-based measures would minimize the influence of case-mix variations and provide clear strategies for the improvement of low-quality ICUs, they contended.

—Jessica Dziedzic

Reference
1. Kahn JM, Kramer AA, Rubenfeld GD. Transferring critically ill patients out of hospital improves the standardized mortality ratio: a simulation study. Chest. 2007;131: 68-75.

Return to table of contents