What Traceability and Sigma-metrics share in common: A Need for Good Data

Posted by Sten Westgard, MS

Tucked into last year's paper on the Roadmap for Harmonization of Clinical Laboratory Measurement Procedures [Clin Chem 57:8; 1108-1117 (2011)] was an interesting discussion of the data available to laboratory professionals on the performance of methods:

• "Peer-reviewed scientific publications using panels of well-characterized patient samples.
• Interlaboratory comparisons that use commutable samples provided by proficiency testing (PT)/ external quality assessment (EQA) schemes.
• Independent reports or reviews submitted or undertaken as part of regulatory approval for market, and
• Reports from postmarket surveillance."

So this  is the data we need in order to properly evaluate the traceability and performance of our methods. Unfortunately, the paper quickly notes, our available data is actually not that good:

"Scientific publications must be scrutinized to ensure that the data reported are valid, i.e., that good experimental design was followed, particularly in relation to the number and quality of patient samples, and the number of different CLMPS included....Data from PT/EQA schemes and other interlaboratory comparisons can be used only when the samples are commutable with clinical patient samples... and of clinically relevant concentrations. Data from noncommutable PT/EQA samples will provide misleading information. Data submitted as part of a regulatory approval process are likely to have been obtained under conditions that may be more strictly controlled for clinical variables than those encountered in routine clinical laboratories, whereas postmarket surveillance generally relies on reports from individual users and may not be representative."

In other words, most of the data we have available to us is wrong. The studies might not have a good design (they may report only a within-run imprecision estimate, for example, instead of the better intermediate ("total") imprecision estimate). The EQA/PT results are generally from programs that use samples that are not commutable, so the bias reported may only be a matrix effect. The studies published in conjunction with regulatory clearance might be overly optimistic (i.e. carefully idealized conditions). The postmarket surveillance studies may only be anecdotal, not systematic and reliable. We have numbers but we might not be able to create a true picture of performance.

There is a well-worn phrase used to address new doctors at their medical school graduations: "Half of what you learned in medical school is wrong. The problem is, we don't know which half."

In the laboratory, we actually know what data is wrong. The bigger problem is, we accept it. It's cheaper and more convenient to accept poor data, so we do.

In order to progress toward the world of Traceability, we need better data that can give us a true idea of the problems in performance with our methods. This is true of Sigma-metrics as well. In order to get a good estimate of performance (the Sigma-metric), we need reliable data on imprecision and bias.

Both Traceability and Sigma-metrics have to be built on a foundation of good data. Regardless of the debate on "uncertainty vs. total analytical error" this is something we can agree on.