Guest Essay

Are molecular diagnostics immune from quality problems? Do they need Quality Control? In this new field of testing, methods and manufacturers have been asserting their "difference" from traditional testing practices, while moving slowly on developing any new and different quality practices. Guest author Dr. Clark Rundell explains that "Traditional" QC protocols can be adopted to address gaps in quality assurance for molecular testing. [Reprint from IVD Technology magazine]

Reprinted with grateful permission from IVD Technology magazine, November 2008 issue.

By Clark A. Rundell, Ph.D.

• Materials and Monitoring
• QC Challenges for Multiplex Tests
• Detecting and Preventing Errors
• Sourcing Control Materials
• Legislation and Regulations
• Conclusion
• References
  

Compared with other laboratory disciplines, the state of the art in quality control (QC) practices for
molecular diagnostic tests has fallen behind. QC for molecular diagnostic tests encounters the following challenges:
new and rapidly evolving technologies, high expectations of accuracy for once in a lifetime genetic tests, lack of quality control materials, lack of quantitative test system outputs, and the almost daily appearance of new genetic test targets. In the face of such issues, clinical laboratories are struggling to develop appropriate quality assurance programs for the molecular diagnostic tests they conduct.

In addition, legislation and government policies threaten to increase oversight of genetic tests to a level not seen before in laboratory medicine (e.g., bills proposed by Senators Edward Kennedy (D-MA) and [ed. then-Senator-] Barack Obama (D-IL), FDA’s guidance on in vitro diagnostic multivariate index assays (IVDMIA), and the Secretary’s Advisory Committee on Genetics, Health, and Society (SACGHS) recommendations).[1] The accuracy of molecular testing is currently unknown, but concerns about quality in molecular diagnostics can only be addressed once data about such accuracy are available. While monitoring molecular test system outputs and statistical analysis provide a good way to obtain data on accuracy and precision, such traditional QC strategies have been slow to take root in molecular diagnostics.

Materials and Monitoring

The laboratorians’ experience and knowledge, the availability of QC materials, the capabilities of test systems,
test costs, and regulatory requirements have an impact on current QC practices. With the earliest molecular tests
that were done manually, the results were determined by interpreting the presence or absence of bands on a gel. Such testing was often performed by researchers with minimal experience in traditional QC techniques, and the QC
materials consisted of previously tested patient samples. Monitoring for systematic errors was deemed to be unnecessary because test failure was easily detected. Furthermore, since the QC materials were variable, labs did not know if a change in band intensity meant an enzyme was degrading or the selected patient control samples did not contain enough DNA. Once failure occurred, labs were required to troubleshoot and retest all of the patient samples. Even though the cost of doing so was a day’s work, the reagents were inexpensive, and the system was easy to troubleshoot.

Molecular test systems are now more complex and expensive. Since samples for genetic molecular tests cost
at least $40 each, retesting is a significant problem. The traditional QC practice of using homogeneous QC materials
could be useful for molecular diagnostics in identifying degrading or defective system components before an actual
test failure. Faced with the limited commercial availability of QC materials, some labs also pool patient samples to create a reproducible source of such materials.[2] Furthermore, new molecular systems have quantitative outputs, such as fluorescence (signal strength) or allelic ratio, which can be tracked to reflect test system performance. Statistical analysis of the QC results over time can establish expected variations. Such results can then be serially plotted on Levy-Jennings charts to monitor the test system for shifts or trends. Westgard rules can also be applied to determine when a corrective action should be taken to prevent test failure.[3]

QC Challenges for Multiplex Tests

More problematic than costs, undetected errors can occur due to the complexity of new multiplex test systems,
even when the results continue to be yes/no answers. The test results are automatically calculated by algorithms determined by the system software. The algorithms provide a quantitative output that takes into account the unique reaction of the probes, primers, enzymes, and background for each allele reported. However, one system component in a multiplex test could skew the algorithm such that an incorrect genotype call is made with no error flags. In addition, only one or two of the reported alleles may be sensitive to a particular system component while the others are sensitive to a different component. When monitoring only a few of the reported alleles in a run, incorrect test results could go undetected.

In each patient sample run, current QC practices for multiplex tests involve testing two to three QC samples,
each representing at most two alleles. When possible, the QC samples containing different alleles should be rotated
so that over time, most mutations assayed in the test will be represented. However, this practice does not ensure that all alleles of a multiplex test are being detected correctly in each run. Traditional QC practices could be adopted by using pooled patient samples of similar genotypes to monitor tests (e.g., thrombotic risk) that detect two or three
alleles. But monitoring a cystic fibrosis 23-plex test for the accurate detection of all mutations is not possible with
only patient samples. In this case, traditional QC practices require homogeneous multiplex controls. While two commercial controls are currently available for cystic fibrosis assays, many more multiplex controls are needed.

Even more problematic are microarray tests or large multiplex test panels that predict diseases or therapies. It
is difficult to imagine a QC material designed to monitor tests that report 100 or more results per sample. Array manufacturers make efforts to provide valid results by building in redundancy. However, such redundancy does not ensure accuracy and does not provide a method to monitor and analyze all system components over time during routine use. Even though traditional QC practices may need to be modified for array test systems, the same following principles apply: adopting traditional QC practices to multiplex tests along with automation of QC statistical analysis may guide the development of QC for complex microarrays.

Detecting and Preventing Errors

A CDC report found that molecular diagnostic test error rates are largely unknown.[4] The limited amount of
proficiency testing (PT) data for molecular diagnostics provides the only information available for assessing laboratory errors. The 2002 U.S. PT data and 2001 European Union PT data showed that 0.1-3% of laboratories had errors on molecular diagnostics proficiency samples.[5,6] According to the 2006 and 2007 PT data from the College of American Pathologists (CAP; Northfield, IL) programs, up to 2% of laboratories had analytical errors for molecular thrombotic risk analysis, or an MTHFR test, on certain proficiency samples. For multiplex cystic fibrosis screening tests, about 4% of laboratories had analytical errors on some proficiency samples. An experimental proficiency study involving a multiplex control with rare genotypes showed a higher error
rate.[7] The causes of the errors in this study included failure to detect a mutation, polymorphisms causing interference with detecting a particular sequence, data misinterpretation by laboratorians, and various errors in reporting the results.

The low number of participants in these studies and the incompatibility of some proficiency samples with certain
molecular test technologies hamper the conclusions to be drawn from this PT data. Moreover, manufacturers of
molecular test systems have been unable to make comprehensive precision and accuracy claims because they lack
sufficient homogeneous QC materials for many of the alleles their systems are designed to detect. As a result, data
to predict the magnitude of errors occurring in molecular diagnostic testing is insufficient.

Current QC practices for genetic tests are limited in their ability to detect and prevent errors. Tests for rare mutations may never be subject to quality assessment, therefore an error prevention strategy or even an estimateof error rates for detecting those alleles is not possible. For more common alleles in which quality controls are routinely run, plotting and analyzing the data for error prediction is not usually practiced. Also, rotating 2-3 controls per run does not generate sufficient data about the accuracy of detecting genotypes not included in the controls.Table I lists examples of traditional quality resources and practices used in clinical chemistry laboratories compared with those used for molecular diagnostic tests.

To address the QC gaps in molecular diagnostic quality assurance, protocols can be adopted from routine clinical
chemistry laboratories. Traditionally, the quantitative data generated from testing homogeneous control samples is
monitored and statistically analyzed for shifts and trends to detect potential test performance problems before failure
occurs. With the exception of high volume virology testing, this approach is new to molecular diagnostics  laboratories. However, thought leaders and early adopters of molecular testing are beginning to use such protocols.[2]

Laboratories have typically turned to published documents from sources such as the Clinical and Laboratory
Standards Institute (CLSI; Wayne, PA) and Westgard QC (Madison, WI) for guidance on validating new tests and
establishing QC protocols.3,8 Although no CLSI documents are currently focused specifically on QC protocols
for molecular tests, several standards and guidelines could still be useful, including the following: “User Protocol for
Evaluation of Qualitative Test Performance (EP12-A2)” and “Verification and Validation of Multiplex Nucleic Acid
Assays (MM-17A).” EP12-A2 presents important theories and statistical techniques for evaluating qualitative tests.
MM-17A contains guidelines for validating multiplex tests and describes several sources for reference materials.

Collecting data for application in QC protocols is being facilitated by numerical system outputs included in most
of the newer molecular test systems. Such software built in by an IVD manufacturer for collecting and displaying QC
data is beneficial for molecular diagnostics, especially for a 23 allele test requiring possibly 69 data charts to monitor
all wild type, heterozygote, and mutant signals. The GeneXpert by Cepheid (Sunnyvale, CA) is an example of a
molecular test system with QC data collection and display capabilities. Having built-in software to analyze statistically QC data and determine action points by Westgard rules and other techniques is a key factor for improving QC of molecular tests.[3]

Sourcing Control Materials

However, the lack of control materials for molecular tests has hindered the adoption of traditional QC methods.
In response, some companies have begun developing and producing controls for genetic testing. Also, even though
an increasing number of national and international sources of characterized cell line reference materials are often not
homogeneous sources of QC, they serve to validate a test system’s ability to detect specific genetic variations.[9,10,11] For example, the CDC Genetic Testing Reference Materials Coordination Program maintains a list of sources for molecular diagnostics quality controls and reference materials.

Despite these resources, the majority of molecular tests continue to lack quality controls useful for traditional QC
practices. Historically, QC is developed only after new tests are implemented and somewhat standardized. Understandably, most IVD manufacturers prefer to sell test platforms that are high ticket items. More so than in other laboratory disciplines, molecular test methods are continuously changing, making it difficult for controls manufacturers to develop products compatible with current and future test systems. Furthermore, the response to current commercial multiplex controls has been lukewarm. Laboratories using free patient samples, or no samples at all, for rare mutations are reluctant to pay for controls. The value of monitoring qualitative tests to prevent failure has not been demonstrated for molecular tests. The test results for all alleles are not monitored, and therefore the failure rates are unknown.

Legislation and Regulations

The Clinical Laboratory Improvement Amendments (CLIA) was written in response to concerns about laboratory
quality. According to CMS, CLIA covered 203,939 laboratories in December 2007, and an estimated 3% of
those labs could conduct molecular diagnostics testing.[12] Molecular diagnostics laboratory directors have the same responsibilities as other directors under CLIA to ensure that test systems provide results of adequate quality for their intended use. CMS has deemed CAP an inspection authority, which provides a molecular pathology checklist of specific quality related activities and the documentation required for certification. Recent recommendations by CDC and SACGHS for increased oversight of molecular diagnostics and tighter requirements for molecular PT suggest future regulations in this area.[1,4]

FDA also plays a role in regulating molecular diagnostics, including the authority to regulate laboratorydeveloped
tests (LDT). To date, the agency has chosen via enforcement discretion not to regulate LDTs. However, while declaring no intention of regulating individual LDTs, FDA has decided to regulate IVDMIA devices, complex
assays with results based on nontransparent algorithms, some of which are developed for in-house use by a single
clinical laboratory. An 18 month phase-in period of regulating IVDMIAs will begin soon. This will be the agency’s
first foray into regulating LDTs.

Laboratories’ resources are already strained, and they would prefer not to have increased regulation. A better
approach to ensuring quality is the voluntary implementation of CLSI guidelines. New CLSI guidelines specifically
detailing the application of traditional QC principles and statistical techniques to molecular tests would be most
ideal.

Conclusion

The following elements are needed to bring molecular QC practice up to the same level as other laboratory disciplines: determining error rates, availability of test system outputs useful for monitoring each test system, adopting
traditional QC protocols to monitor system performance in order to prevent failure, QC materials useful for generating data for system monitoring and error prevention, increased proficiency requirements and samples for molecular tests, and built-in software to facilitate QC strategies.

Due to uniquely difficult challenges, good QC practices for molecular diagnostics have taken longer to evolve than other laboratory disciplines. Unfortunately, inaccurate genetic test results can have serious repercussions for patients and their families. However, by continuing to work together, the IVD industry and the laboratory community can improve molecular QC practices to promote good medicine and avoid burdensome legislation.

References

1. Secretary’s Advisory Committee on Genetics, Health, and Society (SACGHS), “U.S. System of Oversight of Genetic Testing: A Response to the Charge of the Secretary of Health and Human Services,” April 2008; available from Internet: http://www4.od.nih.gov/oba/sacghs/reports/SACGHS_oversight_report.pdf
2. S Liang et al., “Application of Traditional Clinical Pathology Quality Control Techniques to Molecular Pathology,” Journal of Molecular Diagnostics 10, no. 2 (2008): 142-146.
3. JO Westgard, Basic QC Practices, [ed. 3rd edition]. (Madison, WI: Westgard QC Inc., [ed. 2010]).
4. Centers for Disease Control and Prevention, Division of Laboratory Systems Working Group, “Review of Proficiency Testing Services for Clinical Laboratories in the United States: Final Report of a Technical Working Group,” April 2008; available from Internet: www.futurelabmedicine.org/Reports/2007_PT_Report_080320_rev_FINAL.pdf.
5. Centers for Disease Control and Prevention, National Office of Public Health Genomics, “Draft Genetic Test Review: Cystic Fibrosis Analytic Validity,” November 2007; available from Internet: http://www.cdc.gov/genomics/gtesting/acce/FBR/CF/CFAnaVal.htm.
6. E Dequeker et al., “Quality Control in Molecular Genetic Testing,” Nature Reviews Genetics 9, no. 2 (2001): 717-723.
7. S Berwouts et al., “Evaluation and Use of a Synthetic Quality Control Material Included in the European External Quality Assessment Scheme for Cystic Fibrosis,” Human Mutation, May 9, 2008.
8. JO Westgard et al., “Combined Shewhart-cusum Control Chart for Improved Quality Control in Clinical Chemistry,” Clinical Chemistry 23, no. 10 (1977): 1881-1887.
   
9. Centers for Disease Control and Prevention, “Genetic Testing Reference Materials Coordination Program,” November 2007; available from Internet: http://www.cdc.gov/dls/genetics/rmmaterials/default.aspx.
10. National Institute of Standards and Technology, “Standard Reference Materials Catalogue,” September 2008; available from Internet: srmors.nist.gov/tables/view_table.cfm?table=105-8.htm.
11. EuroGentest, “Quality Control and Reference Material Producers Websites,” April 2008; available from Internet: www.eurogentest.org/web/info/public/unit1/reference_materials/rm_databases.xhtml#databases
12. Centers for Disease Control and Prevention, “Laboratory Medicine: A National Status Report,” May 2008; available from Internet: http://www.futurelabmedicine.org/reports/chapter_ii_-_market_profile.pdf.

About the author:

Clark A. Rundell is vice president for research at Maine Molecular Quality Controls Inc. (Scarborough, ME). He can be reached at This email address is being protected from spambots. You need JavaScript enabled to view it..