Risk Management Essays
EQC is out and QC Plans are in
The CLIA Final Rules of 2003 put into place a set of scientifically unsound practices known as Equivalent QC. For years since that time, CMS has been seeking a solution. Now that CLSI has published the EP23 guideline for Risk Analysis, is the successor to EQC at hand?
It’s official: EQC is out and QC Plans are in!
James O. Westgard, Sten A. Westgard
December 2011
With the publication of EP23A[1], there is now a big question: what will happen to EQC? Will it be replaced, modified, or continue as is?
In the November 2011 news story in Clin Lab News, it was stated that “CMS now plans to evaluate EP-23, in concert with other lab accreditors, to consider making risk management a part of CLIA’s interpretive guidelines.”[2]
A CLSI webinar scheduled for December 20th, 2011, “Laboratory Quality Control based on Risk Management” listed Judith Yost from CMS as a key participant. That’s probably a good signal that a policy decision had been made regarding the future of EQC. We contacted CMS by e-mail to ask whether CMS was going to accept QC Plans based on Risk Analysis as “Alternate QC” and also asked what was going to happen with EQC in the future?
We received the following reply: “We plan to adopt many of the concepts contained in EP-23, but we have not yet fully determined to what extent. In any case, this will be gradually phased in and EQC will be phased out. The regulatory requirement will be the default. The choice is voluntary. Obviously with something new, we need to provide education to laboratories and surveyors, etc., prior to making it fully effective. We are working closely with subject matter experts and CLSI on this project.”
A new memo available from the CMS website states the new policy more clearly:
“Initial Plans and Policy Implementation for Clinical and Laboratory Standards Institute (CLSI) Evaluation Protocol-23 (EP), ‘Laboratory Quality Control Based on Risk Management’, as Clinical Laboratory Improvement Amendment (CLIA) Quality Control (QC) Policy.” [3]
You can access this directly on the CMS website (Go to CMS.gov, Survey & Certification, Check the Section Policy and Memos to States and Regions), or access a copy on our website that we’ve stored in the downloads section.
Main Points:
- CMS has formally announced its policy decisions on EQC and QCPs to regional and state agencies. It states that CMS will adopt EP-23 as a QC option that satisfies CLIA QC and that EQC will be phased out.
- CMS will provide a phase-in period of at least 2 years for education and training of its own staff, laboratory inspectors, and accrediting organizations.
- CMS will work with CLSI to provide education and training for laboratories in the application of EP-23 to develop Risk Analysis (RA) Quality Control Plans (QCPs).
- At the time when QCPs are accepted as an approved QC option under CLIA, EQC will be discontinued.
- The default QC requirement is still 42 CFR 493.1256(d)(3): minimum of two levels of external QC per each day of patient testing if EP23 is not used.
The Good News!
The lack of scientific validation of the EQC protocols is finally addressed by elimination of the EQC option as a way of satisfying the CLIA QC requirement. For those laboratories that implemented EQC, this may come as bad news because they must now apply EP23 if they are to provide an acceptable rationale for reducing the frequency of QC.
CMS and CLSI are emphasizing that EP23 allows laboratories to customize their QC Plan to perform the Right QC. “EP23 is not intended to necessarily reduce QC requirements as did EQC, but it is intended to ensure a much more effective QCP for each laboratory and the tests it performs.”
The CLN article quotes Greg Cooper, one of the key participants in the EP23 committee and the CLSI effort at building Risk Analysis guidelines: “Frankly, there are some tests—and everyone in the lab knows which ones those are—that are rock-solid performers that never, ever change. So, do you really need to do external QC every day for those? Probably not. But there are many other tests that are not quite so rock-solid in their performance, and they’re sensitive to reagent lot changes and other variables. For these tests, there is a need to have more intense scrutiny of your QC.”
Read those two quotes carefully. EP23 is not a guarantee that you can reduce your QC frequency for all tests. Some tests – with good performance – can reduce their QC. Other tests may actually need to increase the frequency and intensity of QC.
That’s a good sign. You’ve got a useful management tool if it can discriminate between good and bad performance and deliver an objective judgment. If you’ve got a technique that only tells you what you want to hear, it’s not a tool, it’s just a rubber stamp.
The Right QC
We have long argued for doing the Right QC and provided guidance for selecting SQC procedures on the basis of the quality required for a test and the precision and bias observed for the method [4-6]. For Statistical QC procedures (SQC), the Right SQC involves selecting the right control rules and the right number of control measurements. Guidance for selecting the Right QC can be found in CLSI’s C24A3 guideline [7]. The appendix of that document provides a “Sigma-metric QC Selection Tool” that shows the rejection characteristics of different control rules and different number of control measurements. The probability for rejection is shown on the y-axis vs the size of the systematic error on the y-axis at the bottom and the value of the Sigma-metric on the y-axis at the top. Sigma is calculated for the defined allowable Total Error (TEa) and the observed bias and imprecision of the method [Sigma = (%TEa-%Bias)/%CV].
Unfortunately, the CLIA default of 2 levels of controls is the Right QC only for measurement procedures that achieve 5-sigma performance (or higher). If process performance is less than 4.5-sigma, the Right SQC requires more than 2 control measurements and/or tighter control limits, such as 2.5s. At 4-sigma, the Right QC should employ multirules with Ns from 4 to 6. Below 3.5-sigma, the Right SQC is not economically practical in many laboratories. Here’s where the use of Risk Analysis to develop QC Plans becomes necessary.
The Bad News
It is not a trivial undertaking to employ Risk Analysis for developing QC Plans. Doing Risk Analysis right will require laboratories to expend considerable time and effort to learn a new quality tool and to deploy that tool in an effective way. On one hand, Risk Analysis is intuitive and the idea is easily understood. On the other hand, Risk Analysis is very complicated to apply in a quantitative way, particularly to characterize the expected detection of non-standard control mechanisms and the residual risk of the overall QC Plan. Integration of Six Sigma concepts with Risk Analysis can provide a more quantitative approach [7], but deployment requires additional study and expertise.
Laboratories concerned with ISO 15189 must also remember that QC is to be designed to verify the attainment of the intended quality of test results. That is easily done with SQC procedures whose error detection characteristics are well documented. For non-SQC procedures, manufacturers may have information on their error detection characteristics of their built-in controls, but they are not required to disclose that information to their customers. According to EP23, customers should request that information from manufacturers and it is assumed the manufacturers will then provide it. But what if manufacturers don’t supply that information – or don’t supply a complete picture of the risks of their method or instrument? The assumption that laboratories will be able to have access to all the necessary Risk Information is one of the hidden weaknesses that may limit the practical value and effectiveness of the EP23 approach.
The Road Forward
For better or for worse, QC for the Future is being defined as Risk Management. Unfortunately, many interpret that to mean a fundamentally different approach for developing quality systems, rather than an evolution of past practices. QC has always been about risk, with a specific focus on measuring, monitoring, and managing the variation of a process. Six Sigma Quality Management added a broad understanding of the need for tolerance limits to describe the quality required for a process or product. Sigma-metrics combine that requirement for quality with the performance observed for process variation to provide a measure of quality on the sigma-scale. Such a Sigma-metric can be related to the size of errors that must be detected by QC procedures, thus SQC rules and numbers of control measurements can be readily selected to provide the requisite QC based on the quality required for the test and the precision and bias observed for the method.
SQC should be an essential part of any QC Plan because it can be easily designed to provide detection of medically important errors. For many other control mechanisms in a QC Plan, the detection characteristics are NOT known, therefore there is no way of knowing whether medically important errors will be detected. Qualitative QC Plans won’t satisfy the ISO 15189 requirement to verify the attainment of the intended quality of test results. Therefore, the safest approach for medical laboratories is to make sure they implement the Right SQC as part of their QC Plans.
References
- CLSI EP23A. Laboratory Quality Control based on Risk Management. Clinical Laboratory Standards Institute, Wayne, PA, 2011.
- A New Approach to Quality Control? Clinical Laboratory News, November 2011 http://www.aacc.org/publications/cln/2011/november/Pages/ANewApproachtoQualityControl.aspx
- https://www.cms.gov/Surveycertificationgeninfo/downloads/SCLetter12_03.pdf
- Westgard JO. Basic Planning for Quality. Madison WI:Westgard QC, 2000.
- Westgard JO. Six Sigma Quality Design and Control: Desirable precision and requisite QC for laboratory measurement processes, 2nd ed. Madison WI:Westgard QC, 2006.
- Westgard JO. Assuring the Right Quality Right: Good Laboratory Practices for Verifying the Attainment of the Intended Quality of Test Results. Madison WI:Westgard QC, 2007.
- CLSI C24A3. Statistical Quality Control for Quantitative Measurement Procedures. Clinical Laboratory Standards Organization, Wayne, PA, 2006.
- Westgard JO. Six Sigma Risk Analysis: Designing Analytic QC Plans for the Medical Laboratory. Madison WI:Westgard QC, 2011.