Sigma Metric Analysis
A direct enzymatic assay for %HbA1c
We take a look at a new direct enzymatic method for %HbA1c.
 The Precision and Comparison data
 Determine quality requirements at the critical decision level
 Calculate Sigma metrics
 Summary of Performance by Sigmametrics chart and OPSpecs Chart
 Conclusion
November 2008
[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.] 
This article looks at a new "Direct enzymatic assay for %HbA1c for human whole blood samples" by Limin Liu, Stefanie Hood, Yuping Wang, Robert Berverkov, Chao Dou, Abhijit Datta, Chong Yuan, Clinical Biochemistry 41 (2008) 576583.
Remember, for HbA1c, the critical decision level is 7.0%. That's the crucial cutoff for determinations, diagnosis, and treatment choices. It's also becoming one of the key threshholds for payforperformance programs in diabetic management.
The Precision and Comparison data
Imprecision Estimates:
Two patient samples, one normal (5.7%) and one abnormal (10.3%) , were used to determine total imprecision using the study design from the CLSI EP5A protocol. With 80 data points, total imprecision for both normal and abnormal levels was calculated at 1.8%.
Sample 
Mean

Total Imprecision

Normal Control 
5.7

1.8%

Abnormal Control 
10.3

1.8%

Given the stable estimate of 1.8%, we will assume that the method will also have 1.8% imprecision at a level of 7.0% HbA1c.
Bias estimates (added to the right):
A method comparison study was done with a set of whole blood samples. The Diazyme Direct Enzymatic HbA1c assay was compared to the Tosoh G7 HPLC method and the Roche Tinaquant II immunoassay method.
Comparative Method 
N

Slope

YInt

r2

Tosoh G7 HPLC 
66

0.973

0.111

0.98

Roche Tinaquant II immunoassay 
66

1.10

0.421

0.97

Ideally, the bias is calculated at the critical decision levels for test. Here, we will calculate the bias at the critical level of 7.0%
Remember that the correlation value (r2) is not the important statistic. It dies provide us with this useful information, however: if the correlation is higher that ) 0.975, then simple linear regression is adequate to provide us with good estimates of the systematic error. When correlation is lower than 0.975, it is recommended that a better regression technique should be used, such as a Deming regression or PassingBablock regression.
The authors of this study do not explain whether simple linear regression or Deming regression or PassingBablock was used to calculate the slope and yintercept.
We will assume that the slope and yintercept values that are provided are appropriate.
Calculate bias at the decision level
Now we take the comparison of methods data and set the equation to the level covered in the imprecision study. Solving those equations will give us a bias estimate.
Here are the steps for calculating bias:
((slope*level) + YIntercept)  level) / level = % bias
((0.973*7.0) +0.111)  7.0) / 7.0 = ((6.811 +0.111)  7.0) / 7.0
(6.922  7.0) / 7.0 = 0.078 / 7.0 = 0.0111 * 100 = 1.84%
The bias is expressed as a percentage of the level 7.0%, as an absolute value.
Comparative Method 
N

Slope

YInt

bias%

Tosoh G7 HPLC 
66

0.973

0.111

1.11%

Roche Tinaquant II immunoassay 
66

1.10

0.421

3.99%

Determine the quality requirements at the critical decision level
Now that we have both bias and CV estimates, we are almost ready to calculate the Sigma metrics for these analytes. The last (but not least) thing we need is the quality requirement for each method. CLIA provides quality requirements for over 80 analytes, but in this case, it does not provide a quality requirement.
Source 
Quality Requirement

CLIA PT 
No quality requirement given

CAP PT 2007 
Target value ± 15%

CAP PT 2008 
Target value ± 12%

NACB 2007 Draft 
"interassay CV<5%
(ideally <3%)" 
Clinical Decision Interval 
Target value ± 14%
(biologically based) 
The details of these sources and quality requirements are discussed in Dr. Westgard's essay. The important thing to note here is that there is a pretty big difference between the requirements. Note also that the NACB guidelines do not specifically state any analytical quality requirement  at best, you can infer the the quality requirement based on their specifications for instrument performance. Finally, remember that while the Clinical Decision Interval quality requirement is almost the biggest number, using that number requires that the process take into account the known withinsubject biological variation, which eats up a large amount of the error budget.
Calculate Sigma metrics
Now we have all the pieces in place. What makes this different this time is that we're going to calculate two metrics, one for the bias estimate from the HPLC method, and one from the immunoassay method. Then, we can apply different quality requirements to the calculations. This will produce a dizzying variety of metrics.
Remember the equation for Sigma metric is (TEa  bias) / CV:
For a 12% quality requirement, with the HPLC method, (12  1.11) / 1.8 = 6.05
The metrics are displayed along the right columns.
Direct enzymatic assay compared to HPLC method


Source 
TEa%

Biologic Variation % 
CV%

Bias%

Sigma metric 
CLIA PT 
??

n/a

1.8%

1.11%

???

CAP PT 2007 
15.0%

n/a

1.8%

1.11%

7.72

CAP PT 2008 
12.0%

n/a

1.8%

1.11%

6.05

NACB 2007 Draft 
??

n/a

better than "ideal" specification

1.11%

???

Clinical Decision Interval 
14.0%

4.1%

1.8%

1.11%

4.71

Direct enzymatic assay compared to immunoassay method


Source 
TEa%

Biologic Variation % 
CV%

Bias%

Sigma metric 
CLIA PT 
??

n/a

1.8%

3.99%

???

CAP PT 2007 
15.0%

n/a

1.8%

3.99%

6.1

CAP PT 2008 
12.0%

n/a

1.8%

3.99%

4.45

NACB 2007 Draft 
??

n/a

better than "ideal" specification

3.99%

???

Clinical Decision Interval 
14.0%

4.1%

1.8%

3.99%

3.11

Most of what we see here is good news. Regardless of which method, HPLC or immunoassay, method performance according to CAP standards is good or world class. If we hold the test method to the higher standard of the clinical decision interval (a more "evidencebased" approach), then we should consider ways of improving performance.
Summary of Performance by Sigmametrics chart and OPSpecs chart
Not only can we use tools to graphically depict the performance of the method  we can also use those tools to help determine the best QC procedure to use with that method. Using EZ Rules 3, we can express the method performance on an OPSpecs (Operating Specifications) chart :
Note that this depiction uses the tighter CAP requirement of 12%, and using the performance specifications of 1.8% CV with 1.1% bias vs. HPLC method. The CV and bias become the x, y coordinates of the method's Operating Point. The rule of thumb for an OPSpecs chart is that you want to be as close to the origin of the graph as possible, and if your operating point is below and to the left of the lines on the chart, the QC procedures (shown in the key at the right of the graph) that are represented by those lines are acceptable. In this case, any of the QC procedures shown on the graph would provide adequate error detection  even 3s or 3.5s control limits with 2 controls per run.
Here's a different depiction of performance  a Sigmametrics chart:
In this case, the performance is literally "off the charts"  usually a bold, straight vertical line is drawn in the graph area  and where it intersects the power curves is where the QC procedure performance can be determined. The Sigmametrics chart provides more specific numbers to the error detection and false rejection (see the key at right) as well as the specific Sigmametric of the method.
If you want to visually assess the larger bias against the immunoassay method, you can add another QC Design to the same EZ Rules 3 file. Or you can just look at the existing OPSpecs chart and mentally move the ycoordinate of the operating point up to 3.99. The rise in bias definitely eliminates most of the QC procedures displayed on the chart.
To assess the performance at the 15% quality requirement, you could set up another EZ Rules 3 file, or add another QC Design to the same file, or you could plot all the data points on a Normalized OPSpecs chart.
One last look  apply the clinical QC design model to the data and you get this chart:
Using 1.8 for CV and 1.1 for bias, you've still got many different QC procedures to choose from. The operating point is below 6 different QC procedures  there is even one QC procedure that would use just two controls. This means that the method performs well even when the more demanding constraints of clinical use and biological variation are taken into account.
Conclusions
This application shows that at least some methods are advancing in their performance  and will be able to meet CAP's tighter performance requirements.
This also shows the importance of determining the best reference method for your comparison studies. The bias figures that result from different reference/comparison methods are significantly different. You need to use your best judgment in determing the appropriate way to determine bias for the method when it is in your laboratory.