Sigma Metric Analysis

Continuing in our series of CLIA 2025 check-ups, we take a look at an example Atellica in the US. Can it pass the new goals for CLIA? Or is January 1st going to be the start of an agonizing year?

An Atellica in the US - Can it hit CLIA 2025 targets?

November 2024
Sten Westgard, MS

The new CLIA PT rules went into legal effect in July 2024, but the Proficiency Testing (PT) programs don't have to implement them in their surveys until January 1st, 2025. That provides another half year of anxiety, or respite, depending on your perspective.

Since the airing of our GAMECHANGE webinar, (which is still available on-demand), we've worked with dozens of laboratories to get a glimpse of the January 1st 2025 future. Using Sigma-metrics, we've estimated the quality of laboratory assays and compared them against the key benchmark: 2.3 Sigma.

Why 2.3 Sigma and not 3 Sigma, the traditional minimum allowed quality? Because CLIA 2025, while tightening the acceptability criteria, still relies on a score of 80% for passing success. That is, a lab can fail 20% of the survey and still pass PT. Therefore, the key metric with an assay is 2.3 Sigma, which is the equivalent of a 20% failure rate. If your assay is failing 20% of the time to achieve the results within the CLIA 2025 criteria, that's the danger zone.

Calculating Sigma metrics from routine laboratory performance data.

For this example, we're going to keep the laboratory anonymous, but to give it more relevance to our readers, we'll name a name. This is a Siemens Atellica instrument we're benchmarking, from a laboratory in the US. Using data on cumluative imprecision and cumulative bias against the peer group, we can extract the data needed to benchmark performance on the Six Sigma scale.

[If you need a refresher on Sigma-metrics, check that out here. In this current section of the website, there are hundreds of real-world examples to review.]

Graphic Display of performance by Normalized Method Decision Chart (NMEDx)

42% of the performance measured here is 3 Sigma, 2 Sigma, or less than 2 Sigma. That's not great.

There are some assays where it isn't surprising that there's going to be PT trouble: CO2, sodium, urea, these were challenges before CLIA 2025 and remain challenges after (the sodium goal didn't even change - it's been causing problems for decades now). But there are some new assays where the high or low control level are dropping into the danger zone. Calcium, ggt, magnesium, phosphorous, potassium, triglycerides, are all worrisome in performance. The HbA1c assay is problem, even though the goal is 8% rather than the CAP/NGSP-preferred 6%.

What happens next depends a lot on the configuration of specimens for each survey. If the survey tests a low level where an assay is vulnerable (low sigma), but then the next survey only tests the high levels (where high sigma is more likely), the lab may experience intermittent failures without truly failing proficiency testing. But if there's a run of bad luck - testing the vulnerable part of the range twice in a row - this lab, this instrument, is more likely to fail and fail in a significant way.

How does this lab compare to other US labs?

Again, through the work of GAMECHANGE volunteers, we've been able to benchmark multiple labs and instruments from every major diagnostic manufacturer.

You decode this heat map this way. You want to see Green, Dark Blue, and Blue; that represents the percentage of the assessed menu that achieved 6, 5, or 4 Sigma performance. The Yellow represents 3 Sigma performance (minimum acceptable performance analytically, but passable for PT). The Red represents the percentage of the menu that falls below 3 Sigma, and thus represent unacceptable performance as well as heightened risk for PT failure come January 1st.

Here we can see a snippet of lab-anonymous "Sigma Heat Map" of other Siemens instruments and labs. The column in the center is the performance of this example, and you can see 3 labs to the left, which represent better performance, and another 4 labs to the right, showing worse performance. This laboratory is near the median performance of Siemens instruments, suggesting that most Siemens customers will experience higher risk of proficiency testing failures in 2025. 

Notice also that there isn't as much green in the columns as you have seen with other Sigma heat maps. Another sign of lesser performance.

Want to see more of this? Curious to know where YOU stand with CLIA 2025?

We continue to work with laboratories across the US to assess and benchmark their performance against CLIA's new PT rules. We've many more Siemens performance examples (and Roche and Abbott and QuidelOrtho examples), but those are available to participants in our national benchmarking program. Email This email address is being protected from spambots. You need JavaScript enabled to view it. if you're interested in joining this program. If you submit your performance data, you'll get back a custom sigma assessment, FREE, as well as the full graphs comparing your performance to the rest of the US diagnostic landscape.