Interviews

An interview with CAP about their proficiency testing practices.

For the past year, we've been performing an ongoing analysis of proficiency testing results, looking for data and evidence of test performance and quality. This series has been described in the following articles:

We've had the fortune to make use of public data available from many proficiency testing providers. But as we make calculations and conclusions based on that data, we haven't heard from "the horse's mouth." So we decided to ask some of the proficiency testing providers what they think of their data. The College of American Pathologists was gracious in providing timely and thoughtful answers.

Q: Are Proficiency Testing results indicative of quality in an individual laboratory?

CAP: Proficiency Testing (PT) is a basic component of a well-defined laboratory quality management plan. PT measures the ability of a laboratory to use its instrument or method proficiently, and is one of many laboratory improvement tools that can be used to assess quality. It provides insight into measurement processes that can be translated into improved performance and enhanced measurement quality. In addition to proficiency testing, internal and external quality control, laboratory inspection and accreditation, and quality assurance monitoring provide important tools for measuring laboratory performance using multiple parameters.

Q: While PT is mandated by many regulations, another question remains. Does quality in laboratories improve because of proficiency testing?

CAP: CAP proficiency testing allows each laboratory to compare its test results to the most appropriate peer group, other laboratories that use the same or similar instruments and methods. Comparison of performance to a robust, statistically valid peer group is essential to identify areas for improvement. Areas of improvement that may be identified in a single PT event or over multiple events include: variation from peer group results, imprecision, and/or results that trend above or below the mean consistently or at specific analyte concentrations. Use of this data allows laboratories to continuously improve their test performance.

Retrospective analysis of proficiency testing data has demonstrated that overall performance for a test does improve over time.

Q: How do proficiency-testing results indicate quality in all laboratories?

CAP: Proficiency testing captures a snapshot of analytic test performance for a given analyte at a specific point in time. This information, when used with a complete laboratory quality management plan, forms the basis for providing the optimum outcome…quality patient test results.

Internal quality control results alone cannot adequately identify all types of analytical variance. The use of interlaboratory comparison, provided by PT, allows laboratories to ensure that their instruments are performing at a level comparable to their peers. Laboratories that have a well executed quality management plan, which supports pre-analytical, analytical, and post-analytical accuracy, are more likely to demonstrate a higher level of performance on PT and report accurate patient test results.

Q: Can you explain why CAP PT results are consistently superior to many of the other PT groups – at least in our examinations thus far – to most other PT organizations? Is there a bias toward labs of a certain size and capability?

CAP: The CAP is widely known as the gold standard for laboratory improvement programs. The CAP has over 50 years of experience providing interlaboratory comparison programs. Not only was the CAP the first PT provider, it has a long history prior to the establishment of legislative requirements for educational programs designed to improve the quality of laboratory testing. In fact, today’s legislative requirements are based on some of the components of the CAP Surveys Program.

CAP PT is designed by member pathologist and clinicians that are experts in their areas of specialty. Since these individuals deal with laboratory issues on a daily basis, they are uniquely qualified to provide support for the CAP PT programs.

The CAP provides the most comprehensive array of programs that reach from general hematology to the most complex molecular testing. The offerings provided in each of these areas of testing ensure laboratories an external benchmark for performance evaluation and improvement throughout the laboratory.

CAP PT materials are the highest quality materials available in the industry. Clinically relevant specimen concentrations, strict control procedures, and the inclusion of education challenges designed specifically to address areas for improvement contribute to the value these programs provide.

The most robust, statistically valid database in the industry provides unparalleled support for comparison of results to laboratories using the same instrument or method. Without an adequate peer group comparison, external assessment and improvement is not possible.

In addition to the robust statistical information provided with CAP PT, education activities, supported by member experts, are included without additional charge. The focus of these educational components is to provide information to enhance the knowledge base of personnel and improve the accuracy of test results. Incorporation of information regarding best practices is provided to assist laboratories in staying current with the most recent changes and technologies. These activities also provide continuing education credit for completion and support the continuous improvement philosophy of the CAP.

During the last 50 years, and continuing today, the CAP continues to improve its PT programs to exceed customer expectations and to improve the quality of laboratory medicine.

Q: How are your PT samples prepared and handled? (Chemistries as an example) Liquid samples, fresh patient samples?

CAP: The CAP strives to provide PT challenges that resemble patient specimens as closely as possible to ensure a robust and fair evaluation of performance. In those instances where liquid materials are available, they are preferred. Human based specimens are used when available. Our material vendors adhere to strict specifications, control and qualification procedures, and shipping requirements. Materials must be evaluated and meet the CAP’s criteria before they are accepted into the program. Our material vendors adhere to Good Manufacturing Practice, and in some cases ISO certification, to ensure that the CAP consistently delivers high quality specimens for testing.

Q: How do you handle PT failures by lab? Single, multiple.

CAP: PT results are evaluated for each test analyte for each testing event. The laboratory receives an evaluation report that includes the result they provided and peer comparison information. Historical performance information may be provided in the evaluation to help laboratories analyze their performance over a period of time and monitor their results for deviations prior to the point at which patient care would be compromised.

In addition, a participant summary is provided which contains a summary of all participant results by analyte and defined peer group (instrument/method). With this information a laboratory can evaluate and improve their performance. The CAP also provides additional specimens and information to customers who request support to troubleshoot unsuccessful performance.

Q: Do you monitor control lot issues, instrument issues? Do you communicate regularly with manufacturers on the performance of their controls and methods?

CAP: For the most adequate peer group comparison, instrument and method information is collected from participants in addition to test result information. This data is summarized and forms the basis for analytic comparisons of performance. Participants compare their performance to the most appropriate peer group and take corrective action as necessary. Summary information is available for all instruments and methods in the participant summaries and can be used by customers and manufacturers alike, to evaluate performance.

The CAP is continually working with manufacturers to include new instruments and methods on the appropriate program master lists. In addition, the CAP works cooperatively with manufacturers to evaluate PT material compatibility with their methods and instruments.

Q: Can you give a detailed description of how the results are treated once you receive them? How are averages calculated? What limits do you use to declare a value “out of control” for mean calculation purposes? Do you use manufacturer suggested limits and ranges, or calculate ranges another way?

CAP: On February 28, 1992, the Secretary of Health and Human Services published the final rules implementing the Clinical Laboratory Improvement Amendments (CLIA) of 1988. These regulations established evaluation criteria limits for many of the analytes in the CAP PT programs. The target values are determined by the scientific resource committees of the CAP. For those analytes not included in the PT portion of CLIA, the target values and evaluation criteria are determined solely by the scientific resource committees. With the support of professional biostatisticians, the most statistically robust and innovative grading schemes are used for those analytes not included in the PT portion of CLIA.

For most analytes, the peer group mean is designated as the target value for evaluation. The group must consist of greater than nine results and demonstrate acceptable variability before it is used as the target group. If peer group data are not available or are too variable, method group statistics may be used. If peer group and method group statistics are not available, a comparative method group may be designated as the target mean. If no comparative method exists for the analyte, results will not be evaluated.

For quantitative data, results are grouped according to the method used for analysis and screened for outliers. Various statistics are calculated from the remaining data that summarize the peer group responses. The summary statistics include:

• • Mean (the average of the reported results)
  • Standard deviation (a measure of the variability of the participant results)
  • Coefficient of variation (CV)
  • Median (the middle value in an ordered list of the non-outlier results)
  • Low value
  • High value
  • Final count of reported results that were not excluded as outliers

Outlier exclusion is necessary because a large series of results may include some aberrant values. These may arise from instrument malfunction, technical errors, reversal of vial values, misplaced decimals, incorrect units of measure, or data entry errors. If any results are excluded, the outlier pass is repeated using the remaining values.

For qualitative results, consensus agreement of referee or participating laboratories is used for evaluation. Generally, 80 percent agreement is required.

Rigorous testing of data sets is done on an ongoing basis to assure that grading schemes remain robust and is performed by a professional team of biostatisticians in cooperation with the members of the scientific resource committees.

We thank the CAP for taking the time to answer these questions.