Tools, Technologies and Training for Healthcare Laboratories

Basic QC Practices

Lab Quality: Puzzle or Mystery?

Recently, the two Westgards  did an all-day workshop for the CLMA Tidewater chapter in Williamsburg, Virginia. The theme of the workshop was "Quality Management: Putting the Puzzle Pieces Together." That got us to thinking...

Sten Westgard, MS

2010-Williamsburg-JOWSAW1

From left to right: James O. Westgard, a participant from LabCorps, Nieva Ladrillono, the workshop organizer for CLMA, and Sten Westgard

Our June 25th, 2010 workshop had the title, "Quality Management: Putting the Puzzle Pieces Together," partly because we covered so many topics it was a challenge to connect them together at the end of the day. We had a wide-ranging agenda that covered everything from basic concepts such as Levey-Jennings charts, "Westgard Rules" and method validation concepts, to more advanced face such as Sigma-metrics, Risk Analysis, and FMEA. We covered a lot of ground in a single day. Absorbing and integrating all of that information was undoubtedly a challenge for everyone.

But the title of the workshop got me to thinking, Is quality in the laboratory really a puzzle or is it a mystery?

This question emerges from something I recently read in Malcom Gladwell's What the Dog Saw [1], a collection of his essays for The New Yorker magazine. In one essay, titled Open Secrets[2], Gladwell makes a distinction between puzzles and mysteries, using examples from  the headlines:

  • A puzzle is a question that can be solved by more data. Questions like, Where is Osama bin Laden hiding? are a puzzle. A single critical fact can provide the answer. When Woodward and Bernstein broke the Watergate story, they were solving a puzzle.
  • A mystery, on the other hand, is "a question that cannot be answered with certainty even in principle." [3] What would happen to Iraq after the US toppled Saddam Hussein, for example, was a mystery, despite a wealth of information and experts. More information would not have helped solve the problem, because understanding the context was beyond the ability of any decision-makers.

QC – A Puzzle

For many people in the laboratory, quality control seems like a confusing mystery. They are deluged by "false rejects", which cause them to repeat controls. The repeat-repeat-repeat cycle consequently degrades their confidence in the quality system - if you constantly repeat "false" out-of-control results, can you be sure you're not simply missing a "true" out-of-control result?

Often these concerns are accompanied by additional problems caused by "Westgard Rules", where certain errors happen - for example, a 10x violation - even during a period of time when the bench technologists "know" that the instrument is operating correctly. Thus, the quality control system seems to generate a lot of false alarms, and the technologists are overwhelmed by the effort to keep the controls "in", while in the back of their mind their anxiety grows that the system may never be able to detect a true alarm. How do you really know when a "false reject" is a true reject?

In most of those cases, we can turn those mysteries into puzzles - and provide laboratories with a way to put those pieces together. When labs use 2 sd control limits, for example, that generates a lot of false rejections even when the method is operating correctly. And when labs apply "Westgard Rules" on methods that are highly automated, have excellent performance, and have relatively large quality requirements, the fact that a 10x rule has been violated may not in fact be relevant.

The missing piece of the QC puzzle is often a Quality Requirement. Other times, the missing puzzle piece is the process we call QC Design or QC Planning.  By establishing quality requirements, a laboratory can determine the Sigma-metric of a method. After assessing method performance, the lab can then optimize their QC procedure so that it minimizes false rejections while maximizing error detection. What does that mean? The lab lowers the number of false alarms, while at the same time ensuring that the QC rules will detect the true medically important errors.

For many labs, determining the quality required by a method is a new task - previous to that, the focus has been on regulatory compliance (just do whatever the latest rules say to do)  rather than determining the necessary method performance for clinical needs (do what we should do for good patient care). Quality requirements, Sigma-metrics, and QC Design are new pieces that help solve the puzzle.

But that's not to say that every problem in the laboratory is simply a puzzle waiting to be solved. There are still mysteries in laboratory medicine, indeed in healthcare as a whole.

Medical Decisions – A Mystery

Malcolm Gladwell notes that some current medical tests only generate mystery. In the case of prostate cancer, the diagnosis used to come from a physical exam, but now there are biopsies and PSA tests and ultrasound images. But the added benefit of these tests is questionable:


"Much of that flood of information, however, is inconclusive: elevated levels of PSA don't always mean that you have cancer, and normal levels of PSA don't always mean that you don't - and, in any case, there's debate about what constitutes a normal PSA level.  Nor is the biopsy definitive: because what a pathologist is looking for is early evidence of cancer - and in many cases merely something that might one day turn into cancer - two equally skilled pathologists can easily look at the same sample and disagree about whether there is any cancer present. Even if they do agree, they may disagree about the benefits of treatment, given that most prostate cancers grow so slowly that they never cause problems." [4]

The result of all this uncertainty is that a PSA test may not solve a puzzle, but instead create a mystery. Current advice is that for some men, "watchful waiting" is the recommended action (inaction?) to take upon learning the results of the test.

Other areas where mysteries continue to exist in laboratory medicine include: how do clinicians actually interpret the results of a test? Anecdotally, I know that physicians like to invoke the "art of medicine" clause when it comes to test results. Given the exact same lab report, Dr. Smith may decide you need more treatment, while Dr. Jones may decide you're just fine. Having highly idiosyncratic customers means that laboratories find it difficult to give useful advice on interpretation of results. The fear of contradicting clinician conclusions has lead many a lab to avoid providing advice with test results, and instead focus on areas where it's easier to satisfy the clinician consumer, such as the speed of testing. If we could get physicians to agree on how to interpret test results, labs could provide more useful test reports. But how do we get all the doctors to agree with each other ? That is indeed a mystery.

Harmonized Test Results – A Puzzle or A Mystery?

A similar challenge faces laboratory methods that are not standardized or harmonized. When the same test has multiple method principles and competing, proprietary techniques, the result is often that the test result depends heavily upon the instrument where the test is performed. Take a look at proficiency testing summaries and you'll see the scope of the problem. While clinicians and hospital administrations believe that “a glucose is a glucose is a glucose,” the truth is that for many analytes, significant biases exist between methods.

In theory, this lack of agreement between methods is only a puzzle. We simply need one puzzle piece: the true value for a particular method. However, it's really not so simple. While standard reference methods have been established for some analytes, that has not been achieved for many more tests. Which leads to an existential question such as, How can you determine the true value of a method if a standard doesn't exist, or if two or more equally valid method principles exist for the same test?  An additional complicating factor is that there are incentives for diagnostic companies to resist standardization and instead retain their distinctive methodologies. There is a bit of a contradiction at work: we want standard test results, but market competition between diagnostic manufacturers drives them to differentiate themselves. For example, companies focus on developing unique methods. If a method is proprietary (patented) to a specific diagnostic manufacturer, there are higher financial rewards. A distinct, proprietary method may make more profit than a standardized method that's the same as all other competitors, provided that the diagnostic player holds enough dominance in the marketplace. On the other hand, if diagnostic instruments are all the same (standardized), then testing becomes more and more like a commodity market (assuming - and this is a big assumption - that the quality of all methods are the same), which squeezes profits down over time for all competitors.

Conclusion

Mysteries remain in laboratory testing. When there is no clear cut solution, where a wealth of information seems to overwhelm the ability to comprehend the problem, that's a good sign there is a mystery. But simply because something is a mystery doesn't mean it always will be one.
Many of our laboratory and healthcare mysteries can be turned into "mere" puzzles, with enough effort and coaxing. And one we make our problem into a puzzle, then all that remains is finding the right piece for the solution.

After struggling with mysteries and searching unsuccessfully for missing pieces, some laboratories give up and shun even the attempt at solutions. Rather than trying to comprehend a mystery or solve a puzzle, these laboratories prefer to adopt the compliance route. Compliance never asks more of a laboratory than, Are you doing what the rule says you should do?

It is our hope we're providing you missing pieces for your puzzles, not making the world of the laboratory more mysterious.

References

  1. Malcolm Gladwell, What the Dog Saw, Little, Brown & Company: New York, New York, 2009.

  2. Malcom Gladwell, Open Secrets, January 8, 2007, The New Yorker. To be precise, Gladwell is not the first person to make this distinction, but simply the most well-known writer to popularize the distinction. Gladwell attributes the distinction between puzzles and mysteries to Gregory F. Treverton in Reshaping National Intelligence for an Age of Information, Cambridge University Press (Cambridge, United Kingdom), 2003, p.11-13.

  3. Treverton, op. cit. p.11.

  4. Gladwell, What the Dog Saw, pp.169-170.