Lot-to-lot reagent inconsistencies can have significant effects on patient test results
By Steve Halasey
Variations in the formulation of laboratory reagents have long been known as a source of inconsistent test results. Increasingly sensitive tests and instruments can amplify the problem of inconsistent reagent chemistry, potentially allowing test results to be affected by differences in reagent chemistries.
Labs perform regular quality control activities to detect and address the issue of lot-to-lot reagent consistency. Nevertheless, reagent lot changes and the potential for variability among lots calls for continued vigilance.
To find out how laboratories are dealing with the issues of reagent inconsistency and its potential to affect patient test results, CLP recently spoke with a number of experts in the field.
In Search of Consistency
“What a laboratory needs is consistent performance across reagent lots,” says Greg Miller, PhD, a professor of pathology at Virginia Commonwealth University. “Results for patients should be equivalent when measured with a current and a replacement lot of reagents. Labs need to establish a criterion for acceptable equivalence based on the magnitude of variability in a result that will affect medical decisions made using that laboratory test. Any change in reagent components—that is, raw materials—that alters the equivalence of results should be avoided.”
Variation in the composition of a lab’s reagents is often a cause of issues that can affect a lab’s test results, agrees Ole Dahlberg, vice president and general manager for sample preparation at Thermo Fisher Scientific. Problems resulting from such compositional variations can include changes in the concentration of salts or pH values, precipitates in the solutions, or unit variations for critical enzymes. A related factor in such compositional issues can be “user-caused variation in volumes, mixing, and dispensing,” adds Dahlberg.
Reconstitution error—when a user makes mistakes in preparing a reagent—is among the most frequent causes of reagent inconsistency, agrees Shunika Myles, a technical support representative at Audit MicroControls. Other errors that can result in reagent issues include inconsistent lot recovery, manufacturer instructions that are insufficiently clear for novice technicians or lab assistants, or improper storage that undermines the stability of the reagent, she says.
Storing reagents improperly or at wrong temperatures, or using reagents that are past their expiration date, are also frequent causes of reagent inconsistency, says Jeff Wallace, MS, MT(ASCP), IVD marketing manager in the digital biology group of Bio-Rad Laboratories.
Whatever their cause, lot-to-lot reagent inconsistencies can have significant effects on the results of tests performed in a lab. But labs that perform regular quality control testing should be well equipped to detect problems before they propagate across a large number of test results. According to John C. Yundt-Pacheco, PhD, a scientific fellow in the quality systems division of Bio-Rad Laboratories, labs can protect themselves by watching for any of the following trends that may become apparent during quality control testing:
- An upward or downward shift that affects both controls and patients in a similar fashion.
- An upward or downward shift that affects both the controls and patients but unequally.
- An upward or downward shift that affects controls, but not patients.
Key Types of Failures
If undetected or uncontrolled, however, the effects of reagent inconsistencies can be significant, potentially resulting in shifts in an analyzer’s reported values that may themselves go undetected.
“The key issue is an unknown change in test method performance that gets attributed to changes in the patient rather than the test method,” says Yundt-Pacheco. “Reagent lot crossover studies can identify lot-specific issues when changing lots. But inconsistencies within a reagent lot—where some portion of the reagent packs are different from the others—are much more challenging to detect and rectify.”
“Reagent inconsistency leads to time and effort that is wasted troubleshooting,” says Dahlberg. “This causes a delay in experimentation, because one must wait for new quality reagents to arrive before moving forward.”
Dealing with such inconsistencies and the resulting failures in proficiency testing can require labs to make a number of awkward and time-consuming adjustments, including changes to an analyzer’s calibration protocol, or constantly having to extend or shorten the quality control range of an analyzer, says Myles.
“The main challenge faced by laboratories is what to do when results for patients are not acceptably equivalent when changing reagent lots,” says Miller.“ One option is to reject the reagent lot and get a replacement from the manufacturer. But it could happen that no other lot is available, in which case the results may be unsuitable for patient care decisions. In this case, use of the test should be discontinued and patient samples sent to a referral laboratory until an acceptable reagent lot is available.”
Recognizing Failures Due to Reagent Inconsistency
“Usually the assay will not fail outright,” says Wallace. “Instead you’d likely see an abrupt shift or sometimes a less obvious drift of test control ranges, which would indicate something is wrong with the assay. It could be a problem with the reagents, with the instrument—or both.”
Symptoms of a system spinning out of control can include questionable lab results, QC shifts and trends, out-of-range QC, and proficiency testing errors. “Increases in impurities or changes in chemistry can lead to shifts in limit of detection, false negatives, and potentially false positives,” says Dahlberg.
“The most critical failure mode is an unidentified shift in test method performance that is instead attributed to a shift in patient results,” says Yundt-Pacheco. “If this shift is significant and takes place at clinical decision thresholds, either false positives or false negatives can be generated, causing substantial patient harm.
“A less critical failure mode is an undetected shift in control results, but not the patient results,” he adds. “This will lead to an increase in false rejections and poor QC performance until the QC target is reevaluated.”
Dealing with Inconsistencies
“One way that labs can minimize reagent inconsistency is through agreed-upon specifications and qualification of products with their vendors. This approach is especially useful when the products are critical to operations or will be purchased in large quantities,” says Dahlberg. “Using standard control assays to prequalify reagents upon receipt is another way these issues are being dealt with. Additionally, labs are using proficiency testing on a regular basis to eliminate user variation and update training.”
“Laboratories’ main defense against lot-to-lot reagent inconsistency is the reagent lot crossover study, which evaluates the differences between old and new reagent lots with respect to both patient and QC specimens,” says Yundt-Pacheco. “Interlaboratory peer group programs are another way to evaluate changes in test method performance, though such programs test only quality controls” (Figure 1).
“When labs perform crossover studies with new lots of reagents, whether using assay controls or previous patient samples, they should first determine the number of samples required by calculating the statistical power needed to set acceptable variations,” says Wallace. “Once this range-finding is done, labs can then run the required number of samples with each lot change. Each lab will determine this based on its needs.”
“If the differences with a new reagent lot can be compensated for with adjusted interpretive criteria, then the laboratory must inform clinical users of the test results and provide information about how the difference will affect medical decisions,” says Miller. “If the difference in patient results between two reagent lots is a proportional bias, then a lab can determine a correction factor and apply that correction to the results before reporting them.1
“However, applying a correction factor to an FDA-cleared measuring system puts that assay into the category of a laboratory-developed test, making it subject to the applicable requirements for validating the performance of the test,” Miller observes. “When bias versus the current reagent lot will affect medical decisions, the laboratory is obligated to mitigate the risk of erroneous decisions. A correction is particularly important when medical decisions are being made using fixed decision values from clinical practice guidelines.”
Labs can avoid the extra effort required to validate the performance of a lab-developed test by relying more on quality manufacturing firms with a track record of quality control and quality assurance, says Dahlberg. “As another option, labs can shift to automated systems that reduce or even eliminate human error,” he says.
“Labs can minimize issues related to reagent inconsistency by running suitable control materials at least according to the instrument manufacturer’s instructions, such as one time per shift or per run,” says Wallace. “It’s also essential to follow the instrument manufacturer’s preventive maintenance schedule.”
Laboratories aren’t entirely on their own when it comes to dealing with the issue of reagent inconsistency. A variety of technologies and related protocols are available to help laboratories determine whether there is a problem related to reagent consistency, and to evaluate the extent of the variation.
“Laboratories often use third-party QC materials and trend charts, perform reagent lot crossover studies using QC and patient specimens, and watch closely for error flags in clinical instrument system software,” says Charles Towne, PhD, MB(ASCP), director of technical operations and critical raw materials in the laboratory diagnostics business unit of Siemens Healthineers. “Troubleshooting with a manufacturer is also sometimes necessary. Siemens Healthineers offers services to remotely access a laboratory’s instrument and download detailed testing data, which can help to pinpoint problems or help with troubleshooting during the process of changing reagent lots.”
“Analytical specifications for measured and reported reagents by vendor can help reduce many formulation problems,” says Dahlberg. “By measuring factors such as conductivity, pH, and UV absorbance, using analytical tools such as high-pressure ion chromatography and liquid chromatography–mass spectrometry, labs can generate their own control charts, so that compliance with reagent specifications and lot-to-lot drift can be monitored.”
“Labs should run controls, perform reagent lot crossover studies, and conduct regular instrument QC and preventive maintenance,” says Wallace. “They should also perform quality assurance testing for preanalytical procedures, and monitor test environments on daily basis.
“If a lab suspects variation in its reagents, it should check the reagent manufacturer’s certificate of analysis,” adds Wallace.“If a new reagent lot needs to be calibrated, the lab should make sure that calibration is completed prior to running samples.”
The Clinical and Laboratory Standards Institute (CLSI) offers extensive guidance about how to design and carry out the studies necessary to evaluate reagent lot-to-lot consistency as it pertains to both patient and QC specimens.2 “The studies are structured to determine whether there is an unacceptable amount of variation in the new lot number,” says Yundt-Pacheco. “But while CLSI’s guidance offers a complete solution, it is best implemented with software. To use the guidance, a laboratory should understand the short- and long-term components of variation for each of its test methods.”
Interlaboratory peer group comparison programs can also help to alert participating labs to changes in test method performance with respect to QC, says Yundt-Pacheco (Figure 2). “When the group means and standard deviations change unexpectedly, that’s often an indication that a new reagent lot number with different performance characteristics is being deployed to the field.”
The field experts we spoke with offered mixed views about whether clinical labs are generally prepared to adopt available technologies that can help them to deal with reagent inconsistencies. On the downside, Terry Smith, a technical support representative at Audit MicroControls, says she does not think that labs are currently prepared to adopt such technologies. “Perhaps cost is an issue,” she says.
“Many clinical labs may not have the analytical instrumentation or skills needed to run their own quality control for reagents,” says Dahlberg. “However, adopting good functional testing with control reagents and materials can be implemented in any lab. Our lab routinely uses control samples for nucleic acid preparation and assay development, including the RNA Spike-in Mix developed by the External RNA Controls Consortium, an ad hoc group hosted by the National Institute of Standards and Technology. We also create our own control samples for routine processes.”
Other experts believe that labs are open to adopting technologies to resolve issues related to reagent inconsistency. “To manage their use of reagents, labs often make use of Levey-Jennings control charts, daily QC checks, and cumulative QC reports,” says Myles. “Any solutions that help make workflow easier and deliver quality lab testing and reporting are always helpful and useful.”
“I think labs are open to adopting new technologies,” says Yundt-Pacheco, “particularly when tools are built that automate evaluation processes and don’t require tedious manual steps.”
“All clinical labs should be well-versed in these technologies, and they should have completed proficiency testing as well as individual competency testing to remove those variables,” says Wallace. “Labs should also practice proper training of testing personnel and invite regular inspections by auditing agencies.
“Labs should have a dynamic QA program with a defined set of metrics for regularly inspecting testing departments,” Wallace adds. “They should also use interlaboratory peer group QC programs to check how their instruments and reagents are performing when compared to other laboratories’ instruments and reagents.”
With or without the use of advanced QC technologies, laboratories are required to verify reagent lot performance when changing lots or when receiving a new shipment of the same lot of reagents. Experts advise that laboratories should follow CLSI EP26 or a similar protocol based on comparing results for patient samples between current and new reagent lots.2
When performing a reagent lot crossover study, it is very important not to use results from QC samples measured with both reagent lots to make an assessment of performance for patient samples between the two reagent lots, cautions Miller. “It is well known that processed QC materials have an altered matrix that frequently causes QC results to be different between reagent lots, even when there is no difference in results for patient samples,” he says.3 “This limitation of processed QC materials is addressed in a CLSI guideline, and additional details and approaches to verifying the performance of different reagent lots can be found in several textbooks.4–8
“There are special cases when an in vitro diagnostic manufacturer provides a QC product that is intended specifically for the purpose of verifying acceptable performance of calibrator and reagent lots for that company’s instruments,” Miller adds. “In this case, such an approach is acceptable.”
“As a lab that develops and manufactures reagents for sample preparation, we analyze our reagents with a variety of tools and always follow functional performance with control and real-life samples with a spike in components,” says Dahlberg. “Each lab must decide what are the most important data to collect and track within its cost and time constraints.”
When it comes to protecting the integrity of a lab’s test results from the effects of reagent inconsistency, some test methodologies and analytes are at greater risk than others. Immunoassays, in particular, tend to produce greater variability between reagent lots than other types of tests. But many PCR-based molecular diagnostics are subject to the effects of reagent inconsistencies, and electrolyte test results can also undergo shifts related to the reagent matrix.
“In the grand scheme, I think all tests can be affected by this issue,” says Smith. “In many cases, this is due to not changing calibration protocols and maintaining instruments.”
But while many types of tests may encounter problems related to reagent inconsistency, some test results have greater than average potential for causing patient harm.
“The higher risk measuring systems are for analytes that have decision values for medical action based on small differences in results,” says Miller. “Examples include calcium, glucose, potassium, sodium, and many more.”
Miller offers the example of a tumor marker, such as prostate-specific antigen, that falls to undetectable levels following successful treatment. “Patients are monitored using that tumor marker to identify recurrence that will initiate additional radiation or chemotherapy treatment, and possibly surgery,” he notes. “A small positive bias at low concentrations due to a reagent lot change could result in an erroneous medical treatment, with risk of harm to the patient.”
“Test methods that are used in isolation to form the majority of a clinical decision have a higher risk of patient harm from erroneous results than other, more-general tests,” says Yundt-Pacheco. “Tests that detect cardiac markers, infectious diseases, and tumor markers thus carry greater risk than most biochemistry tests, whose results are usually considered in conjunction with other tests and clinical information to form a diagnosis.”
Incorrect patient diagnoses and their sequelae can often be laid at the feet of much earlier errors, made in the design or handling of reagent components. “Labile analytes such as cardiac markers, parathyroid hormone, and procalcitonin, require the best designs and the most protection to increase material shelf-life,” says Towne. “Among other factors, the shelf-life of materials is dependent on protein stability. Siemens Healthineers maintains traceability of calibrators to a higher-order standard material, in order to maintain accuracy and reduce lot-to-lot variability.”
“Buffers are important if there is a small window of optimal reaction pH,” says Dahlberg. “This can be an issue with samples that are known to have different incoming high or low pH.
“RNA can be a difficult analyte to measure, as it is inherently unstable and can be degraded by both chemical and enzymatic processes,” Dahlberg explains. “Therefore, sample preparation of RNA will be one of the most important steps for determining whether there are targets available to detect. Having ‘clean’ reagents from manufacturing to lab bench is critical.”
What If . . .
Offered a hypothetical situation in which they had identified a significant difference in the performance of two reagent lots, experts describe what they would do to minimize the effects of that difference on patient test results.
Not surprisingly, none of our respondents opted to use reagents from a new lot displaying a significant difference from the previous lot. “I would not use reagents with a lot number that reflects a significant difference,” says Myles.
Some experts offered a more detailed view of their process for handling inconsistencies between old and new reagent lots, often involving a conversation with the reagent manufacturer. “First, the lab should perform instrument maintenance, and then recalibrate the instrument with the new calibrators and reagents,” says Smith. “And then, prior to use, the lab should call the manufacturer.”
“If a significant difference is observed but it is within the accepted specification, then it should be noted and possibly followed up,” says Dahlberg. “If it is outside specification, then different reagents must be used. The standard approach is to use control samples to accurately measure the difference between the lot-to-lot variation.”
“If I deemed the results to be significant but acceptable, I would monitor QC more closely to check for trends developing over time,” says Wallace. “If I deemed the difference to be unacceptable, I would contact the reagent manufacturer. I would also conduct a failure mode investigation of all possible parameters not controlled by the reagent manufacturer to rule out other contributing factors and to ensure that all lab procedures are being followed.”
“If the difference is isolated to the controls, labs can reset their QC targets and ranges,” says Yundt-Pacheco. “But if the difference involves patient results, clinicians need to be notified of the change, with a clear explanation of the differences, and the lab’s reference ranges may need to be reset.”
“A very important responsibility of the clinical laboratory is to provide consistent results for making medical decisions for patient care,” says Miller. “I have on several occasions introduced a correction for bias observed when changing reagent lots. Although this step involves additional validation as a laboratory-developed test, the effort is necessary to maintain consistency of results, so that clinical providers can make appropriate decisions regarding patient care.”
Our experts uniformly confirmed that they would not hesitate to reject a reagent lot based on differences in its lot-to-lot performance. “If a reagent lot is found to be significantly different based on a CLSI EP26 study, it should be rejected if there is a more suitable alternative,” says Yundt-Pacheco.
“Comparing identical reagents from different lots using a set of identical samples should produce results that are within the lab’s acceptable variance,” adds Dahlberg. “If the lab observes results outside of this variance, it must reject the lot. If a swap-in rescues or tightens the variance to an acceptable level, that provides good evidence to demonstrate that a reagent is causing the inconsistent changes in performance.
Experts anticipate that technology solutions will continue to be developed to help labs deal with the issue of reagent inconsistency. But input from laboratories will be essential.
“Laboratories have a role to play in collaborating with IVD manufacturers to develop the requirements for reagents that will produce consistent results, based on medical decisions,” says Miller. “I expect the IVD industry will continue to develop its manufacturing practices to improve the consistency of results between different lots of reagents. There are also important challenges associated with replacement calibrator lots that contribute to consistency of results.”
“Reagent inconsistency can be the result of many influences—including storage conditions and other factors outside of the manufacturer’s control—so there will always be a need to evaluate new reagent lots,” says Yundt-Pacheco. “But I expect the process for evaluating new reagent lots to become more automated and streamlined as this functionality is increasingly built into instruments and middleware.”
“Instrument manufacturers could incorporate internal controls specifically designed to indicate reagent variations,” says Wallace. “This would help manufacturers to reject lots onsite, as part of their lot-release QC.”
“We see automated systems with prefilled and prequalified reagents as the fix for this issue,” says Dahlberg. “Our laboratory has just released a series of products that move sample preparation toward a minimal user interaction experience, such as using prefilled plates and reagents for our Kingfisher Flex and Duo purification systems.”
Experts offered several key pieces of advice for laboratory personnel charged with making sure that patient test results are not adversely affected by inconsistencies between different lots of reagent.
“Reagent inconsistency happens; labs are going to see it, so the best thing they can do is plan ahead,” says Wallace. “Labs need to have systems in place to catch it. They should not try to run on too lean of an inventory, or wait until their inventory is nearly depleted before running lot crossover studies.”
“Some variations among lot numbers are to be expected. However, it would be wise to monitor lots that consistently show a large increase in variation,” says Myles. “If possible, labs should make themselves familiar with the manufacturing practices for the reagent, because increasingly lot variations can become problematic and cause for a change in manufacturer.”
“The devil is in the details,” says Dahlberg. “Labs should not overlook or underestimate the power of variation stacking when running complex assays that use many reagents.
“Labs should be skeptical about the performance of reagent changes until they have evidence that supports the new reagent lot’s suitability,” says Yundt-Pacheco. “It’s appropriate to be wary of matrix effects, where reagent lots behave differently with controls than with patient samples. And labs should become very familiar with CLSI guideline EP26 and set up the infrastructure to rigorously evaluate every reagent change.”
- Measurement Procedure Comparison and Bias Estimation Using Patient Samples. [CLSI guideline EP09c.] Third ed. Wayne, Pa: Clinical and Laboratory Standards Institute, 2018. Available at: https://clsi.org/standards/products/method-evaluation/documents/ep09. Accessed May 1, 2019.
- User Evaluation of Between-Reagent Lot Variation. [CLSI guideline EP26-A.] Wayne, Pa: Clinical and Laboratory Standards Institute, 2013. Available at: https://clsi.org/standards/products/method-evaluation/documents/ep26. Accessed May 1, 2019.
- Miller WG, Erek A, Cunningham TD, Oladipo O, Scott MG, Johnson RE. Commutability limitations influence quality control results with different reagent lots. Clin Chem. 2011;57(1):76–83; doi: 10.1373/clinchem.2010.148106.
- Statistical Quality Control for Quantitative Measurement Procedures. Fourth ed. [CLSI guideline C24.] Wayne, Pa: Clinical and Laboratory Standards Institute, 2016. Available at: https://clsi.org/standards/products/clinical-chemistry-and-toxicology/documents/c24. Accessed May 1, 2019.
- Miller WG, Nichols JH. Quality control. In: Clarke W, ed. Contemporary Practice in Clinical Chemistry. Third ed. Washington, DC: AACC Press, 2016:47–62.
- Miller WG. Quality control. In: McPherson RA, Pincus MR, eds. Henry’s Clinical Diagnosis and Management by Laboratory Methods. 23rd ed. Philadelphia: Elsevier Saunders, 2016:112–129.
- Miller WG, Sandberg S. Quality control of the analytical measurement process. In: Rifai N, Horvath AR, Wittwer C, eds. Tietz Textbook of Clinical Chemistry and Molecular Diagnostics. Sixth ed. Amsterdam: Elsevier, 2017:121–156.
- Miller WG, Sandberg S. Quality management. In: Rifai N, Horvath AR, Wittwer CT, eds. Tietz Fundamentals of Clinical Chemistry and Molecular Diagnostics. Eighth ed. Amsterdam: Elsevier, 2019: 90–107.
Featured image: Laboratory test tubes. Photo by Gina Sanders courtesy Dreamstime (ID 8007061).