James O. Westgard, PhD
Here are some notes, comments, and links that go along with my presentation at the AACC teleconference on “New Directions in Laboratory QC: EQC, Alternate QC and Risk Assessment,” which was on September 4, 2008.
There is a strong push by CMS and CLSI to introduce new QC guidelines that are based on “risk analysis.” This direction is consistent with ISO’s emphasis on risk analysis for manufacturers. Soon we will see how it influences for laboratory QC. I realize that US laboratories may not yet have much interest in ISO, but we need to recognize that ISO will play an increasing role for guidance in quality management in the future.
QC Options. CLIA rules allow laboratories 4 different options for QC. Here I call these options statistical QC (SQC), deFault QC (FQC), Equivalent QC (EQC), and Alternate QC (AQC). They are discussed in the context of the CLIA regulation as well as ISO 15189 guidance for medical laboratories. The reason for including ISO is that the emphasis on risk analysis originates with ISO, thus we should also understand the context in which ISO describes guidance for QC.
Statistical QC. The first option is to do statistical QC (SQC) to monitor the precision and accuracy of the process, detect immediate errors, and monitor test performance over time. This can be done following guidelines from ISO 15189 and CLSI C24-A3, as outlined in the slides. A Sigma-metrics tool will soon be available for download from this website. It will include directions, a Sigma-metrics graph for 2 levels of QC, and a sigma-metrics graph for 3 levels of QC. Note that the C24-A3 document does not include a tool for 3 levels of material. There are many other materials on Sigma-metrics and QC planning on this website. Start with the following if you want more information and background:
Some background on “power curves” – those detector response curves that show the probability of rejecting runs have different sizes of errors – can be found in the lesson “Power Function Graphs” at www.westgard.com/lesson4.htm.
The application of “Sigma metrics” for selecting the “right QC” procedures is described in the lesson “CLIA Final Rule: Appropriate QC Procedures” at www.westgard.com/cliafinalrule9.htm.
There also is a PDF file available for a paper “Internal Quality Control: Planning and Implementation Strategies. www.westgard.com/IQCpaper2003.html
Default QC. There is little to say! This is what most laboratories do today – two levels every 24 hours. However, you should understand that this was never the intention of CLIA that this would become the standard QC practice. It was supposed to be temporary guidance until such time as the FDA provided a QC clearance process that would provide for review and approval of a manufacturer’s QC directions. After five postponements and some 10 plus years of default QC, it was concluded there no longer was a need for an FDA QC clearance process. For some historical background on CLIA and its original recommendation for QC clearance, see the following:
For an explanation of the QC clearance provision, see “QC Clearance Postponed Again & Again & Again & Again.” www.westgard.com/essay37.htm
For a brief summary of the changes in the final, final, final, final, final version of CLIA, see “The Final CLIA Rule.” www.westgard.com/essay50.htm
Equivalent QC. There is a lot of discussion on this website. Start with the following:
See “Equivocal QC: Coming soon to laboratory near you!” www.westgard.com/essay119.htm. At the end of this essay you will find links to the most important earlier discussions. There’s probably more than you want to know, but you may get interested in seeing how this has evolved over the last few years.
Alternate QC. There’s not much specific information available unless you can find drafts of the EP22 and EP23 documents. Those will be hard to come by because they are pretty well protected until the documents are ready to be released. That means the first public drafts will likely become available in 2009 and the final approved drafts probably in 2010. Meanwhile, you can set up a process by which you can take advantage of this information when it becomes available. That’s what I’ve briefly described as a “Total QC Strategy” near the end of this presentation – see slides 34-36 and the following discussions on this website:
“EQC, AQC, and QC Clearance. From Quality to Compliance. From Compliant to Complicit.” www.westgard.com/essay121.htm
“Waiting for the FDA” by Sten Westgard. www.westgard.com/guest38.htm
“Formulating a Total QC Strategy.” www.westgard.com/lesson55.htm
Risk Analysis Too. Be sure to read Dr. Kent Dooley’s essay on “Frequency of QC: A Patient Safety Perspective. www.westgard.com/guest35.htm
What’s SAFE?
A few comments about my thinking here.
SQC is first on my list because we currently have good design tools for selecting the right SQC rules and numbers of control measurements. The main limitation is that we don’t have an objective methodology for determining how often controls should be run. Here’s where Dr. Parvin’s ideas about “event” and “non-event” QC provide the best available guidance.
AQC is lower on the list because we don’t yet have the guidance on how to do this. However, if you consider that SQC will most likely be necessary under most conditions to monitor “residual risks,” then the ability to design SQC sets the stage for what should follow with AQC. It’s possible that AQC will move into the #1 spot if the guidelines turns out to be practical and easy to implement, which will wreck my SAFE acronym. I’m not sure how long that will take, but I’m betting that I’m SAFE for another 2 years and by that time I’ll probably really be retired. Also, those tools in the QC tool box may not be so easy to implement, in which case AQC may never become practical. Patient data QC sounds good and can be very useful, but it’s not trivial to optimize the design of patient data algorithms to assure the attainment of the intended quality of results.
FQC gets an F on my grading sheet. The only plus is that you’re still running controls, but you really should implement the right QC design, which would then move this into SQC above.
EQC is even worse because you’re not running controls as often. Remember that the laboratory director must take full responsibility for any decision to implement EQC. CLIA allows you to do this, but if you do it, you are responsible and you can’t blame CLIA if you run into problems. By that I mean that CLIA will not be a “get out of jail” card here. You do this at your own risk.
Q&A. Some additional information that may provide more complete answers to some of the questions that were asked:
Concerning how to review, assess, and evaluate your QC procedures and process, see “Good Laboratory Practices for Statistical QC. Part I. Designing the Right QC.” www.westgard.com/essay102.htm. It is also very important to implement SQC properly, which is discussed in “Part II. QC Limits and Limitations.” www.westgard.com/essay103.htm
Concerning slide 14 and the figures in column Ped in the key. Those figures should be 0.00. I generate the underlying power curves here using a computer program that typically draws a single line for a particular error condition or sigma-value. Here I need a figure without that line, so I have to fake out the program to get the underlying graphic. It looks like I must have set the error condition to be off-scale on the left, therefore ended up with the same figures as for false rejection (Pfr). Someone has pretty sharp eyes because no one has ever caught me at this before! Now I will have to figure out how to get zeros.
Concerning the question about instruments that have internal statistical control: Yes, there is at least one BGAS system that has a complete internal SQC procedure, as well as many additional built in instrument checks. And that system still must be qualified for EQC according to CMS, even though the internal SQC is much better than any external SQC in this case because of the sample handling difficulties with BGAS controls.
Concerning the use of patient data algorithms, such as “Bull’s algorithm” or the more generic “average of normals” or “average of patients,” see “Six Sigma: Patient Data for Assessing Process Performance and Stability” at www.westgard.com/qcapp17.htm. For some explanation of “multi-stage QC,” i.e., the idea of using multiple QC procedures in a complementary way, see the discussion “Sage Advice about New Approaches to Quality Control” at www.westgard.com/essay30.htm
Compliance vs Excellence.
CAP has introduced a new accreditation program CAP:15189 that is based on the ISO document. CAP says that this doesn’t replace the LAP accreditation which is based on CLIA. And they explain that “although it is not currently a standard in the US, CAP believes that a laboratory that is working to achieve best practices in quality management systems will seek out accreditation to ISO 15189:2007 and its benefits.” It looks to me like we’re heading for a two-tiered system where CLIA will be used for compliance with minimum standards and ISO 15189 will be used for excellence. That’s why we need to start paying attention to ISO 15189 if we want our laboratories to remain competitive in the global marketplace. For additional discussion about CAP and its intentions with 15189, see Sten’s interview with CAP officials at www.westgard.com/interview8.htm.
-----
By Sten Westgard, MS
While Dr. Westgard has spoken about the "War of Words" between Uncertainty and Total Error in several essays (here, here and here most recently), we thought readers might want an additional perspective on the issue.
Dietmar Stockl, an expert in statistics and laboratory quality control from across the Atlantic, graces us with an guest essay on the calculation, use, benefits and limitations of measurement uncertainty: Time to Engage in Measurement Uncertainty. Dr. Stockl provides an in-depth look at uncertainty concepts and calculations, as well as a moderate viewpoint on the use of the term. He views both Total Error and Measurement Uncertainty are useful concepts and believes there is room in the world for both of them. It's not necessary for one term to eliminate the other.
Not so coincidentally, a colleague of Dr. Stockl's, Linda Thienpont, has a letter in Clinical Chemistry talking about the attempt to incorporate bias into the calculation of measurement uncertainty. If bias can be thrown into the uncertainty calculations, a case could be made that Total Error is no longer necessary. Dr. Thienpont points out that slipping bias into the uncertainty calculations is not a good idea and can lead to distorted results. She concludes that bias must remain separate from uncertainty calculations, which means another approach like Total Error is required to account for it.
See more at Thienpont LM. Calculation of measurement uncertainty-Why bias should be treated separately. Clin Chem 2008;54:1587 (subscription required)
-----
Posted by Sten Westgard
While Dr. Westgard has given a thorough update on the status of EP22 and EP23, as presented at the AACC/ASCLS conference in Washington, DC, and Dr. Jan Krouwer has done the same on his own blog, I thought it worth beating the dead horse on this.
Dr. Jan Krouwer was there at the presentation and asked perhaps the most important question about EP22 since it was created. Is the FDA going to review the Risk information created by manufacturers about their instruments?
I think the answer was somewhat nuanced and subject to interpretation. According to Dr. Krouwer, he heard that the FDA would review EP22 Risk information, but only superficially, and "only egregious problems would be flagged by the FDA."
Basically, I heard the same thing, with a slightly different interpretation: the FDA isn't really going to review the Risk information when the device is first submitted to them. But retrospectively, it might review them very thoroughly. Since the FDA has authority over labeling and branding of medical devices, if it turns out that some of the risk information from a manufacturer is incorrect, inaccurate, or misleading, than the FDA retains the power to sanction the manufacturer.
In other words, manufacturers have the right to remain silent (i.e. don't produce any Risk information, since EP22 is voluntary), and anything they say (i.e. the EP22 Risk information) can be used against them in a court of law.
Call it the Manufacturer's Miranda Risk Rights.
In the absence of a legal mandate, I don't see why any manufacturer would willingly create this Risk information and distribute it to their customers. Any corporate lawyer will be advising against it (liability exposure). Any marketer will argue against it (if our competitors aren't saying they have residual risks, why do we need to do so?). Any executive will oppose it (why are we exposing our weaknesses to our competitors and our customers?). And that's the just the internal opposition this will face.
Certainly, I believe disclosure of Risk is the right thing to do, but the competitive and legal environment are not going to be receptive to it. Unless customers overwhelmingly demand this information, manufacturers will have no incentive - and many disincentives - to produce Risk information.
As Dr. Krouwer laments, there is a lot of good work in the EP22 and EP23 guidelines, but because of the toxic environment, these standards are likely to be stillborn.
Yet another sorry milepost in the story of EQC, AQC and Option 4.
-----
posted by Sten Westgard
Yes, the Westgard blog is back.
Regular visitors will remember that back in 2006, Westgard QC dipped its toe in the blog water, but after a dozen or so postings, the blog went silent.
It turns out, the technology we were using to run the blog was not that robust or user-friendly. And it was particularly vulnerable to SPAM, with hundreds, if not thousands, of scripts attempting to comment on the blog with links to frightening websites. That forced us to be very strict on comments so we could be certain only genuine comments could go through. The combination of these problems meant that the blog was really not usable for our purpose. We ceased posting but kept it barely alive. The revamping of the blog went on the To-Do list. Unfortunately, that list was so long it took a few years to get done.
Now, we're using a better blog platform, with more capabilities and features than before. This should allow better posting, better commenting, less SPAM, and easier navigation. (Also note, the previous posts from the old blog have been transferred over, so you still have access to the content.)
The blog is always going to me an adjunct to the main westgard website, www.westgard.com. But we'll use this to point to new articles and events, as well as make shorter comments and points about topics of interest to the laboratory community. There are a lot of issues and news stories that warrant a short note, but aren't really worth an entire essay.
In short, we hope that we'll be able to post more stuff and do it faster.
Thanks for your continued readership of the website and the blog. We hope we can continue to improve its offerings.
Don't forget to make your thoughts known in the comments.
-----