Challenges for the implementation of the in-vitro diagnostics regulations in the light of â??real-lifeâ?? performance of blood glucose self-monitoring devices

Corresponding Author:

Anja Schuster
Department of Biomedical Sciences, Salzburg University of Applied Sciences, Salzburg, Austria
E-mail: anja.schuster@fh-salzburg.ac.at

Abstract

Objective: Accurate and reliable blood glucose monitoring devices (BGMDs) are required for adequate self-monitoring and management for people suffering from diabetes mellitus. To ensure a high level of safety for patients and users of in vitro diagnostic devices, an updated In Vitro Diagnostic Regulation (IVDR 2017/746) has been published, including new safety and performance prerequisites. However, BGMD accuracy may be limited by certain ‘real-life’ environmental factors, which should be respected in the performance evaluation. Methods: The temperature-and humidity-dependent performance of 4 different BGMD using up to 440 capillary blood samples was determined. BGMDs with associated test strips were stored at 15°C, 25°C and 35°C resp. at a relative humidity of 40% and 80% in order to imitate potential ‘real-life’ performance settings. Glucose measurements were compared to blood glucose values determined using standardized Laboratory equipment and were analyzed based on the ISO 15197:2013 system accuracy criteria. Results: Two out of three BGMDs provided consistent results across temperature ranges based on the medical threshold of a mean glucose change less than 15 mg/dl, although blood glucose difference of up to 96 mg/dl was found at an individual patient level after pairwise temperature comparison. For one device up to 31.2% of patients’ values were outside the defined limits when comparing 15º C to 35º C. Changes in humidity levels did not significantly influence the mean values across the BGMDs, although high deviations were observed at patients’ individual glucose levels. Conclusion: Moderate temperature and humidity changes can affect the accuracy of point-ofcare devices to a profound extent at an individual patient level. These ‘real-life’ environmental factors need to be included in the performance evaluation as required in the IVDR 2017/746 in order to provide a solid testing system for novel point-of-care-devices.

Keywords

■ blood glucose monitoring systems ■ device accuracy ■ IVDR 2017/746 ■ humidity ■ temperature ■ ‘real-life’ environmental setting ■ performance evaluation ■ health literacy

Introduction

The global prevalence of diabetes is rising continuously [1] and when no effective prevention methods are adopted, the numbers will raise to an estimated 693 million diabetes patients by 2045 [2]. Diabetes can have lifethreatening consequences if the disease is not managed appropriately. Therefore, strict glucose control and monitoring is recommended for patients, in order to minimize the progression of diabetes and its potential complications. Self-monitoring of blood glucose (BG) plays an important role for safe and efficient therapeutic decisions, especially in insulin treated patients [3]. For this reason, a large number of medical device manufacturers have developed a range of different brands and types of blood glucose self-monitoring devices (BGMDs). In order to ensure a high level of safety for patients and users of In Vitro diagnostic devices, including BGMDs, the European Parliament lays down safety and performance requirements for medical devices used in humans. A new regulation for In Vitro diagnostic devices (IVD), the In Vitro Diagnostic Regulation (IVDR 2017/746) of the European Union [4], has been published on the 5th of May 2017 with a 5-year transition period to implement new requirements on many IVD manufacturers seeking a Conformité Européenne (CE) labelling for their devices.

The new classification system for IVD is based on risk rules and consists of 4 risk categories: A being the lowest risk up to D being the highest risk for patients, health care providers and third parties. BGMDs and the accompanying test strips fall under class C, defined as high personal risk IVD. Being classified as a class C product, technical documentation must be more detailed compared to the past and conformity assessment procedures will require a notified body-involvement in the future, similar to the medical device regulation. Manufacturers will have to conduct a performance evaluation to proof product safety and performance. The evaluation includes scientific validity, analytical and clinical performance as well as continuously monitoring of BGMDs throughout their life cycle with post-market performance follow-up. This new obligatory QM-system also includes a unique device identification, which enables a faster and more efficient product recall.

As mentioned above, in addition to the new requirements stated in the IVDR 2017/746, BGMDs marketed in the European Union require a CE labelling which includes the International Organization for Standardization (ISO) standard EN ISO 15197:2015 as a minimum requirement [5]. The European Committee for Standardization (CEN) is requested to revise the existing standard EN ISO 15197:2015 in support of the Regulation (EU) 2017/745 for medical devices by the 27th of May 2024. Until this date, accuracy requirements for BGMDs are defined as 95% of BG results being within ± 15 mg/dL (0.83 mmol/L) of the reference measurement at BG concentrations <100 mg/dL (5.55 mmol/L) and within 15% at BG concentrations ≥ 100 mg/dL (5.55 mmol/L). Furthermore, at least 99% of all results have to be within the clinically acceptable error grid zones A+B [5].

Compliance with these standards are an essential prerequisite to ensure patient safety and avoid potential life-threatening hyperglycaemia or hypoglycaemia [6]. However, a number of studies have shown that there is a considerable number of BGMDs on the market which do not fulfill the minimum requirements stated in the ISO 15197 [7–10]. A number of factors can influence the analytical reliability of BGMDs, including environmental conditions, especially the prevailing temperature and humidity, as well as rapid shifts in ambient temperature [11–13].

In this respect, the new requirement for clinical evidence is of utmost importance to allow a qualified safety assessment of the device when used as intended by the manufacturer. Nevertheless, diabetes requires active selfmanagement by patients, which presupposes a competence to proficiently “obtain, process, and understand basic health information and services needed to make appropriate health decisions”, also known as health literacy [14].

This study’s aim was to evaluate the effect of changes in environmental factors, including temperature and humidity, on system accuracy of four different BGMDs based on ISO 15197:2015 and to discuss the results in the light of the new IVDR 2017/746 in order to support the development of solid testing systems for novel and already used point-of-care-devices.

Methods

Patient blood samples and ethics

All research efforts conducted on humans followed the World Medical Association’s Declaration of Helsinki. The study was performed in cooperation with the diabetic outpatient laboratory of the Department of Internal Medicine I, Divisions of Nephrology and Endocrinology at the Paracelsus Medical University in Salzburg from which a total of 440 blood samples were retrieved. BG values measured at the outpatient laboratory using the Biosen C Line (EKF Diagnostic) were used as a reference value. Informed consent was obtained from of all participants as part of a larger study (DM2CUA).

Ethics approval: The ethics commission of the county of Salzburg approved the study under the following number: 415-E/2438/3-2018.

Investigational devices

In the study, four different BGMD were used: Accu-Chek® Performa (ACP) and its follow up device Accu-Chek® Guide (ACG) from Roche, CONTOUR® NEXT ONE (CON) from Ascensia Diabetes Care and FreeStyle Freedom Lite (FFL) from Abbott. Additional materials, including test strips (various test strip lots were used) and control solutions, were used according to the information provided by the manufacturer. These devices were chosen since they were at the time of the study the most widely prescribed BGMDs in Austria.

Study procedures

Temperature dependency of BGMD: The temperature dependent performance of three different BGMD (ACP, CON and FFL) using 440 patient capillary blood samples was determined. Over the course of the experiment, which was executed across 22 days, two BGMDs per manufacturer and brand plus associated test strips were stored in an appropriate incubator (Cooling incubator KB 53 E3.1, Binder; LLGuniINCU 20 Digital Mini Incubator, LLG Labware) for each temperature level (15, 25 and 35°C) to ensure controlled conditions and avoid temperature fluctuations. Various lot numbers of test strips were used during the study. Control solutions for each device, with a glucose concentration range specified by the manufacturer, and capillary blood were measured in duplicate with each of the BGMD at all three temperature levels (n=2 × 440 for each temperature). Control measurements using control solutions provided by the manufacturers were carried out daily at ambient temperature to check the system accuracy of the BGMDs.

Effect of humidity on BGMD: The humidity dependent performance of three different BGMDs (ACG, CON and FFL) using 127 patient capillary blood samples was determined. Two BGMDs per manufacturer and brand plus associated test strips were stored in a constant climate chamber (HPP110, Memmert) at a stable temperature of 20°C and a humidity of either 40% or 80% to ensure controlled conditions and avoid humidity and temperature fluctuations. According to a recommendation of Memmert, BGMDs were stored in the climate chamber without batteries. Control solutions and capillary blood from a total of 127 patients were analyzed in duplicate with all three BGMDs at both relative humidity-levels (n=2 × 127). The values detected in the diabetic outpatient laboratory were used as reference values.

Data analysis

All data were imported to IBM SPSS Statistics version 25 (IBM Corporation, NY, US). Statistical analysis regarding temperature- and humidity-dependency were performed using the Wilcoxon Signed Rank test including Bonferroni correction for pairwise comparison within each device (temperature and humidity) as well as between device for humidity data and Kruskal Wallis Test (non-parametric data) and Dunn’s PostHoc test was used for comparison between the devices for temperature data. Analysis was performed based on the sample’s glucose concentrations (<100 mg/dl and ≥ 100 mg/dl) as stipulated in ISO 15197:2013. Data is reported as minimum and maximum change in glucose concentration, the mean change in glucose concentration with standard deviation as well as pairwise comparisons. In addition, the percentage of values outside the range defined by ISO 15197:2015-12 (± 15 mg/dl or ± 15% at glucose concentrations of <100 mg/dl resp. ≥ 100 mg/dl) is given.

Results

Temperature dependent performance based on control solutions

Temperature dependent performance of the three BGMDs was first determined using control solutions for each device with a glucose concentration range specified by the manufacturer under certain conditions. For FreeStyle Freedom Lite (FFL) and Accu-Chek® Performa (ACP) all control measurements were within the specified range at all temperatures measured FIGURE 1.

For the CONTOUR® NEXT ONE (CON), only the control measurements at 25°C were within the range, however at 15°C and 35°C, the measurements at all levels (low, middle and high glucose concentration) were outside the specified range.

Temperature dependent performance of BGMD based on patient samples

Temperature dependency on the performance of the three BGMDs, independent of the lot number which was used, was further determined using samples from 440 diabetes patients. All patient samples were measured in parallel with each of the devices at 15, 25 and 35°C. Direct pairwise comparison between the temperature levels within each device resulted in significantly different values for the devices FFL and CON at all temperatures (p<0.01; except FFL 25/35°C p=0.028) FIGURE 2. ACP did not show any significant differences at any of the temperatures measured. In addition to statistical significance, the actual values and deviations according to the ISO 15197:2015-12 are given in TABLES 1 and 2. Values outside the defined fluctuation rangebetween 4.2% and 31.2% of all measures-gave either too low or too high glucose concentrations. The highest deviation was found for CON comparing temperature levels 15 and 35°C.

TABLE 1. Temperature dependent performance of the three BGMDs. For each pairwise comparison of temperature, the patients are grouped into low (<100 mg/dl) and high (≥ 100 mg/dl) BG concentrations (sorted based on the glucose concentration measured at 25°C). Data is shown as minimum and maximum change in glucose concentration as well as the mean change in glucose concentration with standard deviation. In addition, the percentage of values outside the range defined by ISO 15197:2015-12 (± 15 mg/dl or ± 15% at glucose concentrations <100 mg/dl resp. ≥ 100 mg/dl) is given and printed in bold when mean value is outside this range.

Table 2. Humidity dependent performance of the three BGMDs. For each pairwise comparison between humidity levels, the patients are grouped into low (<100 mg/dl) and high (≥ 100 mg/dl) BG concentrations (sorted on the glucose concentration at 25°C). Data is shown as minimum and maximum change in glucose concentration as well as the mean change in glucose concentration with standard deviation. In addition, the percentage of values outside the range defined by ISO 15197:2015-12 (± 15 mg/dl or ± 15% at glucose concentrations <100 mg/dl resp. ≥ 100 mg/dl) is given.

Further statistical analysis was performed based on the lot number used for each device (FFL- 4 different lots, ACP-3 different lots, CON-2 different lots) FIGURE S1 and TABLE S1. The same trend for temperature dependency was seen for each device lot compared to the mean of all lots per device which indicates consistency between the lots.

To account for device differences, the performance between the three devices at each temperature level was analyzed, based on the whole patient population. At 15°C, no significant difference was observed, however significant differences were reached comparing FFL with CON at 25°C (p<0.0001/Δ 15,5 mg/ dl glucose) and CON with ACP (p<0.0001/Δ 13 mg/dl glucose) as well as CON with FFL at 35°C (p<0.0001/Δ 27 mg/dl glucose). To account for differences between standardized equipment (reference values) and point of care devices, comparison was performed between the patient values measured at the hospital (reference values) at room temperature and each device at 25°C which re sulted in significant differences (p<0.01/FFL Δ 12 mg/dl; ACP Δ 32 mg/dl; CON Δ 37 mg/dl glucose) for all measures. Further details comparing the reference device with the BGMD based on low and high glucose values can be found in TABLE S2.

Humidity dependent performance based on control solutions and patient blood samples

Humidity dependency on the performance of the three BGMD was determined at a humidity of 40% and 80%. Quality control measurements were performed using the control solutions for each device at both humidity-levels. All control measurements were within the specified control range as defined by the manufacturers, indicating no performance deviation dependent on humidity. In addition to the control solution measurements, 127 patient blood samples were used to determine variations in the results based on humidity changes. Statistical analysis of direct pairwise comparison between the humidity within each device resulted in significantly different values (p<0.01) for the devices ACG and CON FIGURE 3.

In addition, we report the minimum and maximum glucose change, the mean glucose changes as well as the percentage of values outside the ISO defined range after pairwise humidity comparison TABLE 2. Even though the results from devices ACG and CON are considered statistically significant and individual glucose values show a high variation all changes are within the requirements stated by the ISO 15197:2015-12 and therefore not considered significant in a medical sense. In addition, only up to 5.9% of the measures were outside the defined fluctuation limit TABLE 2.

To account for device differences, the performance between the 3 devices at each humidity was analyzed and showed significant differences only between the devices ACG and FFL at 80% humidity (p<0.01; Δ 3 mg/dl). However, all changes are within the requirements stated by the ISO 15197:2015-12 and therefore not considered significant in a medical sense.

Discussion

In the scope of this study, we evaluated the performance of 4 frequently used point-of-care glucose test systems that are affected by changes in temperature and humidity with which users have to deal under ‘real-life’ circumstances. Accurate glucose monitoring is critical for appropriate glycemic management, which includes appropriate storage and handling of all test equipment, correct use of the BG meters and consistency and accuracy of the test strips and lots used. In addition, environmental factors, such as temperature, humidity, altitude and electromagnetic radiation, physiological factors as well as medication have a potential impact on the analytical performance of the devices [3]. A brief survey focusing on the dayto- day diabetes management of 20 diabetes patients performed at the Salzburg University of Applied Sciences showed limited health literacy skills and inadequate handling of their device of these patients despite having had a training course [15]. Marciano, et al. acknowledged in their meta-analysis, that “higher levels of health literacy were significantly associated with better diabetes knowledge and lower levels of HbA1C” [16] indicating the importance for the role of health literacy in self-care and glycemic control. The relatively low level of health literacy shows, that the use of the BGMD should be fool-prove in order to ensure reliable measurements. This asks for a system that is stable under all usual circumstances.

Measurement reliability and accuracy of the 4 BGMDs was assessed using a moderate change of temperature, including 15°C, 25°C and 35°C, and humidity-levels including 40% and 80%.

Even at these moderate changes we could identify significant differences in the patient BG readout as well as control solution readouts for BGMDs from certain manufacturers. Overall, ACP/ACG and the FFL provided consistent and reliable results across the temperature and humidity ranges using control solutions as well as patient blood samples (mean glucose changes less than 15 mg/dl). In contrast, CON provided unstable results across the control solution measurements and exceeded the medical threshold of the mean glucose change of 15 mg/dl comparing 15°C to 35°C. Even though the mean changes in glucose values after pairwise comparison are mainly within the defined fluctuation range (except CON), individual changes were shown to be quite substantial in all BGMDs ranging from, e.g. -61 mg/dl to +96 mg/dl for FFL and -70 mg/dl to +99 mg/dl for CON. This is reflected by up to 9% of patient’s values being outside the defined limitation for FFL and ACP and up to 31,2% for CON, whereby higher deviation is seen with blood glucose level (BGL) above 100 mg/dl. With respect to humidity measures, all devices resulted in consistent mean glucose concentrations for both, control solutions as well as patient samples. However, individual glucose levels, reported as minimum and maximum change in glucose concentration, showed a substantial (ACG and CON) deviation comparing the two humidity levels.

Studies investigating the accuracy of BGMD at a range of temperatures from 12°C–35°C and humidity from 49–100% showed that both temperature and humidity had significant effects on the reliability of nearly all BGMDs, whereby the effect of temperature was greater than the effect of humidity [17, 18]. A 15 to 30 minutes acclimatization of BGMDs to room temperature is reported to lessen these effects [19].

Our study highlights the range of variations in BG values based on changes in temperature, humidity and device used. Even though the mean change in BGL for the various comparisons performed where within the accuracy requirements stipulated by International Organization for Standardization (ISO) 15197:2003, the variations per patient are alarming, leading to inappropriate applications of insulin doses, thereby causing potentially life threatening hypoglycemia or hyperglycemia. Up-to-date point-of-care devices should include a warning if the environmental conditions including temperature and humidity are outside the reliable operating conditions with an appropriate health-literacy-sensitive education of the patient upon receipt of their point-of-care device. The newly required ‘in-use’ stability report as stated in IVDR EU 2017/746 Annex II 6.3, shelf life validation as well as post-market follow-up will provide essential information on the ‘real-life’ performance of newly developed BGMD and will considerably contribute to patient safety and protect users of In Vitro diagnostic devices by minimizing intra-and inter-variations of BGMD.

Conclusion

Even moderate temperature and humitidy changes affect the accuracy of BGMDs to a profound extent at an individual patient level. In addition, the results vary depending on the device used. Therefore, precisely defined regulations as required by the IVDR 2017/746 are urgently needed to ensure the accuracy of the BG measurements independent of the device used. In addition, a thorough consultation and continuous training of the patients regarding the handling and maintenance of their BGMD is essential to avoid insulin over- or under-dosing.

Acknowledgements

Part of the data was acquired with help by students in biomedical sciences, we thank Tina Treml and Philip Langwallner for their practical work within this project. In addition, we thank all patients who participated in the study, and finally the nurses in the diabetes out-patient clinic for patient recruitment and support in the organization of this project.

Funding source

This study was funded by a local funding of the county of Salzburg (DM2CUA).

Disclosures

None

Conflict of interests

The Author(s) declare(s) that there is no conflict of interest

Author contributions

G.O., K.S., L.S and R.W. contributed to the design of the study. K.S. and C.R. were involved in the data acquisition. A.S. and C.R. analyzed the data and prepared the first draft of the manuscript. All authors interpreted the results, reviewed and approved the manuscript. G.O. supervised the study.

Abbreviations

Accu-Chek® Performa (ACP); Accu-Chek® Guide (ACG); Blood Glucose Monitoring Devices (BGMD); Blood Glucose (BG); Blood Glucose Level (BGL); CONTOUR® NEXT ONE (CON); FreeStyle Freedom Lite (FFL), In Vitro diagnostic devices (IVD)

References

1. Lin X, Xu Y, Pan X, et al. Global, regional, and national burden and trend of diabetes in 195 countries and territories: An analysis from 1990 to 2025. Sci. Rep. 10(1), 14790 (2020).
2. Cho NH, Shaw JE, Karuranga S, et al. IDF diabetes atlas: Global estimates of diabetes prevalence for 2017 and projections for 2045. Diabetes Res Clin Pr. 138, 271–281 (2018).
3. Erbach M, Freckmann G, Hinzmann R, et al. Interferences and limitations in blood glucose self-testing: An overview of the current knowledge. J Diabetes Sci Technol. 10(5), 1161–1168 (2016).
4. The European Parliament and of the Council of the European Union. Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC(2017/745). OJ. (2017).
5. In Vitro diagnostic test systems-Requirements for blood-glucose monitoring systems for self-testing in managing diabetes mellitus (ISO 15197:2013). Beuth Verlag GmbH, Berlin. (2015).
6. Tamaki M, Kanazawa A, Shirakami A, et al. A case of false hypoglycemia by SMBG due to improper storage of glucometer test strips. Diabetol Int. 5(3), 199–201 (2014).
7. Freckmann G, Link M, Schmid C, et al. System accuracy evaluation of different blood glucose monitoring systems following ISO 15197:2013 by using two different comparison methods. Diabetes Technol The. 17(9), 635–648 (2015).
8. Jeanny G, Hope P. Surveillance of the system accuracy of two systems for self-monitoring of blood glucose after market approval. J Diabetes Sci Technol. 10(1), 240–241 (2015).
9. Yu-Fei W, Wei-Ping J, Ming-Hsun W, et al. Accuracy evaluation of 19 blood glucose monitoring systems manufactured in the asia-pacific region: A multicenter study. J Diabetes Sci Technol. 11(5), 953–965 (2017).
10. Pleus S, Baumstark A, Jendrike N, et al. System accuracy evaluation of 18 CE-marked current-generation blood glucose monitoring systems based on EN ISO 15197:2015. BMJ Open Diabetes Res Care. 8(1), (2020).
11. Deakin S, Steele D, Clarke S, et al. Cook and chill: Effect of temperature on the performance of nonequilibrated blood glucose meters. J Diabetes Sci Technol. 9(6), 1260–1269 (2015).
12. Pratumvinit B, Charoenkoop N, Niwattisaiwong S, et al. The effects of temperature and relative humidity on point-of-care glucose measurements in hospital practice in a tropical clinical setting. J Diabetes Sci Technol. 10(5),1094–1100 (2016).
13. Ginsberg BH. Factors affecting blood glucose monitoring: sources of errors in measurement. J Diabetes Sci Technol. 3(4), 903–913 (2009).
14. Health Literacy: Improving Health, Health Systems, and Health Policy Around the World: Workshop Summary. National Academies Press (US) (2013).
15. Tiefengrabner M, Simon Ginzinger, Grabner B, et al. Diabetes mellitus type 2: Clever individual support in everyday life (DM2CUA) (2018).
16. Marciano L, Camerini A-L, Schulz PJ. The Role of Health Literacy in Diabetes Knowledge, Self-Care, and Glycemic Control: a Meta-analysis. J Gen Intern Med. 34(6): 1007–1017 (2019).
17. Haller MJ, Shuster JJ, Schatz D, et al. Adverse impact of temperature and humidity on blood glucose monitoring reliability: a pilot study. Diabetes Technol The. 9(1): 1–9 (2007).
18. Lam M, Louie RF, Curtis CM, et al. Short-Term Thermal-Humidity Shock Affects Point-of-Care Glucose Testing: Implications for Health Professionals and Patients. J Diabetes Sci Technol. 8(1): 83–88 (2014).
19. Nerhus K, Rustad P, Sandberg S. Effect of ambient temperature on analytical performance of self-monitoring blood glucose systems. Diabetes Technol The. 13(9): 883–892 (2011).