OIV-MA-AS1-05 Principle of validation of routine methods with respect of reference methods

Principle of validation of routine methods with respect to reference methods

The OIV acknowledges the existence of methods of analysis of wines in addition to those described in the Summary of International Methods of Analysis of Wines and Musts, of common methods most often automated.  These methods are economically and commercially important because they permit maintaining a complete and efficient analytical framework around the production and marketing of wine.  Moreover, these methods allow the use of modern means of analysis and the development and adaptation of techniques of analysis.

In order to allow laboratories to use these methods and to insure their linkage to methods described within the Summary, the OIV decides to establish a plan of evaluation and validation by a laboratory of an alternative, common method, mechanized or not with respect to a reference method described in the Summary of International Methods of Analysis of Wines and Musts.

This principle, which will be adapted to the particular situation of the analysis of wines and musts, will take its inspiration from international standards in current use and allow the laboratory to assess and validate its alternative method in two ways:

OIV-MA-AS1-07 Collaborative study

The purpose of the collaborative study is to give a quantified indication of the precision of method of analysis, expressed as its repeatability r and reproducibility R.

Repeatability: the value below which the absolute difference between two single test results obtained using the same method on identical test material, under the same conditions (same operator, same apparatus, same laboratory and a short period of time) may be expected to lie within a specified probability.

Reproducibility: the value below which the absolute difference between two single test results obtained using the same method on identical test material, under different conditions (different operators, different apparatus and/or different laboratories and/or different time) may be expected to lie within a specified probability.

The term "individual result" is the value obtained when the standardized trial method is applied, once and fully, to a single sample. Unless otherwise stated, the probability is 95%.

General Principles

• The method subjected to trial must be standardized, that is, chosen from the existing methods as the method best suited for subsequent general use.
• The protocol must be clear and precise.
• The number of laboratories participating must be at least ten.
• The samples used in the trials must be taken from homogeneous batches of material.
• The levels of the analyte to be determined must cover the concentrations generally encountered.
• Those taking part must have a good experience of the technique employed.
• For each participant, all analyses must be conducted within the same laboratory by the same analyst.
• The method must be followed as strictly as possible.  Any departure from the method described must be documented.
• The experimental values must be determined under strictly identical conditions: on the same type of apparatus, etc.
• They must be determined independently of each other and immediately after each other.
• The results must be expressed by all laboratories in the same units, to the same number of decimal places.
• Five replicate experimental values must be determined, free from outliers.  If an experimental value is an outlier according to the Grubbs test, three additional measurements must be taken.

Statistical Model

The statistical methods set out in this document are given for one level (concentration, sample).  If there are a number of levels, the statistical evaluation must be made separately for each.  If a linear relationship is found (y = bx or y = a + bx) as between the repeatability (r) or reproducibility (R) and the concentration (), a regression of r (or R) may be run as a function of .

The statistical methods given below suppose normally‑distributed random values.

The steps to be followed are as follows:

A/ Elimination of outliers within a single laboratory by Grubbs test.  Outliers are values which depart so far from the other experimental values that these deviations cannot be regarded as random, assuming the causes of such deviations are not known.

B/ Examine whether all laboratories are working to the same precision, by comparing variances by the Bartlett test and Cochran test.  Eliminate those laboratories for which statistically deviant values are obtained.

C/ Track down the systematic errors from the remaining laboratories by a variance analysis and by a Dixon test identify the extreme outlier values.  Eliminate those laboratories for which the outlier values are significant.

D/ From the remaining figures, calculate standard deviation of repeatability); Sr., and repeatability r standard deviation of reproducibility SR and reproducibility R.

Notation:

The following designations have been chosen:

m Number of laboratories

i(i = 1, 2... m) Index (No. of the laboratory)

 Number of individual values from the ith laboratory

Total number of individual values

x(i = 1, 2... ni) Individual value of the ith laboratory

Mean value of the ith laboratory

Total mean value

Standard deviation of the ith laboratory

A/ Verification of outlier values within one laborator

After determining five individual values , a Grubbs test is performed at the laboratory, to identify the outliers’ values.

Test the null hypothesis whereby the experimental value with the greatest absolute deviation from the mean is not an outlier observation.

Calculate PG =

= suspect value

Compare PG with the corresponding value shown in Table 1 for P = 95%.

If PG < value as read, value is not an outlier and si can be calculated.

If PG > value as read, value probably is an outlier therefore make a further three determinations.

Calculate the Grubbs test for with the eight determinations.

If PG > corresponding value for P = 99%, regard as a deviant value and calculate without .

B/ Comparison of variances among laboratories

Bartlett Test

The Bartlett test allows us to examine both major and minor variances.  It serves to test the null hypothesis of the equality of variances in all laboratories, as against the alternative hypothesis whereby the variances are not equal in the case of some laboratories.

At least five individual values are required per laboratory.

Calculate the statistics of the test:

Compare PB with the value indicated in table 2 at m - 1 degrees of freedom.

If PB > the value in the table, there are differences among the variances.

The Cochran test is used to confirm that the variance from one laboratory is greater than that from other laboratories.

Calculate the test statistics:

Compare PC with the value shown in table 3 for m and at P = 99%.

If PC > the table value, the variance is significantly greater than the others.

If there is a significant result from the Bartlett or Cochran tests, eliminate the outlier variance and calculate the statistical test again.

In the absence of a statistical method appropriate to a simultaneous test of several outlier values, the repeated application of the tests is permitted, but should be used with caution.

If the laboratories produce variances that differ sharply from each other, an investigation must be made to find the causes and to decide whether the experimental values found by those laboratories are to be eliminated or not.  If they are, the coordinator will have to consider how representative the remaining laboratories are.

If statistical analysis shows that there are differing variances, this shows that the laboratories have operated the methods at varying precisions.  This may be due to inadequate practice or to lack of clarity or inadequate description in the method.

C/ Systematic errors

Systematic errors made by laboratories are identified using either Fischer's method or Dixon's test.

R .A. Fischer variance analysis

This test is applied to the remaining experimental values from the laboratories with an identical variance.

The test is used to identify whether the spread of the mean values from the laboratories is very much greater than that for the individual values expressed by the variance among the laboratories () or the variance within the laboratories ().

Calculate the test statistics :

Compare PF with the corresponding value shown in table 4 (distribution of F) where = = m ‑ 1 and = = N ‑ m degrees of freedom.

If PF > the table value, it can be concluded that there are differences among the means, that is, there are systematic errors.

Dixon test

This test enables us to confirm that the mean from one laboratory is greater or smaller than that from the other laboratories.

Take a data series Z(h), h = 1,2,3...H, ranged in increasing order.

Calculate the statistics for the test:

Compare the greatest value of Q with the critical values shown in table 5.

If the test statistic is > the table value at P = 95%, the mean in question can be regarded as an outlier.

If there is a significant result in the R A Fischer variance analysis or the Dixon test, eliminate one of the extreme values and calculate the test statistics again with

the remaining values. As regards repeated application of the tests, see the explanations in paragraph (B).

If the systematic errors are found, the corresponding experimental values concerned must not be included in subsequent computations; the cause of the systematic error must be investigated.

D/Calculating repeatability (r) and reproducibility (R).

From the results remaining after elimination of outliers, calculate the standard deviation of repeatability sr and repeatability r, and the standard deviation of reproducibility sR and reproducibility R, which are shown as characteristic values of the method of analysis.

If there is no difference between the means from the laboratories, then there is no difference between sr and sR or between r and R.  But, if we find differences among the laboratory means, although these may be tolerated for practical considerations, we have to show and and r and R.

Bibliography

• AFNOR, norme NFX06041, Fidélitè des méthodes d'essai.  Détermination de la répétabilité et de la reproductibilité par essais interlaboratoires.
• DAVIES O. L., GOLDSMITH P.l., Statistical Methods in Research and Production, Oliver and Boyd, Edinburgh, 1972.
• GOETSCH F. H., KRÖNERT W., OLSCHIMKE D., OTTO U., VIERKÖTTER S., Meth. An., 1978, No 667.
• GOTTSCHALK G., KAISER K. E., Einführung in die Varianzanalyse und Ringversuche, B‑1 Hoschultaschenbücher, Band 775, 1976.
• GRAF, HENNING, WILRICH, Statistische Methoden bei textilen Untersuchungen, Springer Verlag, Berlin, Heidelberg, New York, 1974.
• GRUBBS F. E., Sample Criteria for Testing Outlying Observations, The Annals of Mathematical Statistics, 1950, vol. 21, p 27‑58.
• GRUBBS F. E., Procedures for Detecting Outlying Observations in Samples, Technometrics, 1969, vol. 11, No 1, p 1‑21.
• GRUBBS F. E. and BECK G., Extension of Sample Sizes and Percentage Points for Significance Tests of Outlying Observations, Technometrics, 1972, vol. 14, No 4, p 847‑854.
• ISO, norme 5725.
• KAISER R., GOTTSCHALK G., Elementare Tests zur Beurteilung von Messdaten, B‑I Hochschultaschenbücher, Band 774, 1972.
• LIENERT G. A., Verteilungsfreie Verfahren in der Biostatistik, Band I, Verlag Anton Haine, Meisenheim am Glan, 1973.
• NALIMOV V. V., The Application of Mathematical Statistics to Chemical Analysis, Pergamon Press, Oxford, London, Paris, Frankfurt, 1963.
• SACHS L., Statistische Auswertungsmethoden, Springer Verlag, Berlin, Heidelberg, New York, 1968

Table 6 –  Results of the collaborative study

OIV-MA-AS1-08 Reliability of analytical results

Data concerning the reliability of analytical methods, as determined by collaborative studies, are applicable in the following cases:

1. Verifying the results obtained by a laboratory with a reference method
2. Evaluating analytical results which indicate a legal limit has been exceeded
3. Comparing results obtained by two or more laboratories and comparing those results with a reference value
4. Evaluating results obtained from a non-validated method

1. Verification of the acceptability of results obtained with a reference method

The validity of analytical results depends on the following:

• the laboratory should perform all analyses within the framework of an appropriate quality control system which includes the organization, responsibilities, procedures, etc.
• as part of the quality control system, the laboratory should operate according to an internal Quality Control Procedure
• results should be obtained in accordance with the acceptability criteria described in the internal Quality Control Procedure

Internal quality control shall be established in accordance with internationally recognized standards, such those of the IUPAC document titled, "Harmonized Guidelines for Internal Quality Control in Analytical Laboratories."

Internal Quality Control implies an analysis of the reference material.

Reference samples should consist of a template of the samples to be analyzed and should contain an appropriate, known concentration of the substance analyzed which is similar to that found in the sample.

To the extent possible, reference material shall be certified by an internationally recognized organization.

However, for many types of analysis, there are no certified reference materials.  In this case, one could use, for example, material analyzed by several laboratories in a competence test and considering the average of the results to be the value assigned to the substance analyzed.

One could also prepare reference material by formulation (model solution with known components) or by adding a known quantity of the substance analyzed to a

sample which does not contain (or not yet contain) the substance by means of a recovery test (dosed addition) on one of the samples to analyze.

Quality Control is assured by adding reference material to each series of samples, and analyzing these pairs (test samples and reference material).  This verifies correct implementation of the method and should be independent of the analytical calibration and protocol as its goal is to verify the aforementioned.

Series means a number of samples analyzed under repeatable conditions.  Internal controls serve to ensure the appropriate level of uncertainty is not exceeded.

If the analytical results are considered to be part of a normal population whose mean is m and standard deviation is s, only around 0.3% of the results will be outside the limits m ± 3s.  When aberrant results are obtained (outside these limits), the system is considered to be outside statistical control (unreliable data).

The control is graphically represented using Shewhart Control Graphs.  To produce these graphical results, the measured values obtained from the reference material are placed on the vertical axis while the series numbers are placed on the horizontal axis.  The graph also includes horizontal lines representing the mean, m, m ± 2 (warning limits) and m ± 3 (action limits) (Figure 1).

To estimate the standard deviation, a control should be analyzed, in pairs, in at least 12 trials.  Each analytical pair shall be analyzed under repeatable conditions and randomly inserted in a sample series.  Analyses will be duplicated on different days to reflect reasonable changes from one series to another.  Variations can have several causes: modification of the reactants composition, instrument re-calibration and even different operators.  After eliminating aberrant data using the Grubbs test, calculate the standard deviation to construct the Shewhart graphs.  This standard deviation is compared to that of the reference method.  If a published precision level is not obtained for the reference method, caused should be investigated.

The precision limits of the laboratory should be periodically revised by repeating the indicated procedure.

Once the Quality Control graph is constructed, graph the results obtained from each series for the control material.

A series is considered outside statistical control if:

(I)a value is outside the action limit,

(II)the current and previous values are situated outside the attention limits even in within the action limits,

III) nine successive values lie on the same side of the mean.

The laboratory response to "outside control" conditions is to reject the results for the series and perform tests to determine the cause, then take action to remedy the situation.

A Shewhart Control Graph can also be produced for the differences between analytical pairs in the same sample, especially when reference material does not exist.  In this case, the absolute difference between two analyses of the same sample is graphed.  The graph's lower line is 0 and the attention limit is 1.128 while the action limit is 3.686Sw where = the standard deviation of a series.

This type of graph only accounts for repeatability.  It should be no greater than the published repeatability limit for the method.

In the absence of control material, it sometimes becomes necessary to verify that the reproducibility limit of the reference method is not exceeded by comparing the results obtained to those of obtained by an experimental laboratory using the same sample.

Each laboratory performs two tests and the following formula is used:

= Critical difference (P=0,95)

= Means of 2 results obtained by lab 1

= Means of 2 results obtained by lab 2

R = Reproducibility of reference method

r = Repeatability of reference method

If the critical difference has been exceeded, the underlying reason is to be found and the test is to be repeated within one month.

2. Evaluation of analytic results indicating that a legal limit has been exceeded

When analytical results indicated that a legal limit has been exceeded, the following procedure should be followed:

In the case of an individual result, conduct a second test under repeatable conditions.  If it is not possible to conduct a second test under repeatable conditions, conduct a double analysis under repeatable conditions and use these data to evaluate the critical difference.

Determine the absolute value of the difference between the mean of the results obtained under repeatable conditions and the legal limit.  An absolute value of the difference which is greater than the critical distance indicates that the sample does not fit the specifications.

Critical difference is calculated by the formula:Mean of results obtained

= Limit

n=Number of analyses

R=reproducibility

r=repeatability

In other words, this is a maximal limit where the average of the results obtained should not be greater than:

If the limit is a minimum, the average of the results obtained should not be less than:

3. Comparing results obtained using two or more laboratories and comparing these results to a reference value

To determine whether or not data originating in two laboratories are in agreement, calculate the absolute difference between the two results and compare to the critical difference:= Mean of 2 results obtained by lab 1

=Mean of 2 results obtained by lab 2

= number of analyses in lab 1 sample

=number of analyses in lab 2 sample

R=Reproducibility of reference method

r=Repeatability of reference method

If the result is the average of two tests, the equation can be simplified to:

If the data are individual results, the critical difference is R.

If the critical difference is not exceeded, the conclusion is that the results of the two laboratories are in agreement.

Comparing results obtained by several laboratories with a reference value:

Suppose p laboratories have made n1 determinations, whose mean for each laboratory is y₁ and whose total mean is:

The mean of all laboratories is compared with the reference value.  If the absolute difference exceeds the critical difference, as calculated using the following formula, we conclude the results are not in agreement with the reference value:

=Critical difference, calculated as indicated in point 2, for the reference method.

For example, the reference value can be the value assigned to a reference material or the

value obtained by the same laboratory or by a different laboratory with a different method.

4. Evaluating analytical results obtained using non-valitated methods

A provisional reproducibility value can be assigned to a non-validated method by comparing it to that of a second laboratory:

= Mean of 2 results obtained by lab 1

= Mean of 2 results obtained by lab 2

r = Repeatability of reference method

Provisional reproducibility can be used to calculate critical difference.

If provisional reproducibility is less than twice the value of repeatability, it should be set to 2r.

A reproducibility value greater than three times repeatability or twice the value calculated using the Horwitz equation is not acceptable.

Horwitz equation:

%=Standard deviation for reproducibility(expressed as a percentage of the mean)

C= concentration, expressed as a decimal fraction (for example,   10g/100g = 0.1)

This equation was empirically obtained from more than 3000 collaborative studies including a diverse group of analyzed substances, matrices and measurement techniques.  In the absence of other information, RSD_R values that are lower or equal to the RSD_R values calculated using the Horwitz equation can be considered acceptable.

 values calculated by the Horwitz equation:

If the result obtained using a non-validated method is close to the limit specified by legislation, the decision on the limit shall be decided as follows (for upper limits):

and, for lower limits,

S = decision limit

= legal limit

= provisional reproducibility for non-validated method

=reproducibility for reference method

= critical difference, calculated as indicated in point 2, for the reference method

The result which exceeds the decision limit should be replaced with a final result obtained using the reference method.

Critical differences for probability levels other than 95%

This difference can be determined by multiplying the critical differences at the 95% level by the coefficients shown in Table 1.

Table 1 - Multiplicative coefficients allowing

the calculation of critical differences for

probability levels other than 95%

Bibliography

• "Harmonized Guidelines for Internal Quality Control in Analytical Chemistry Laboratories". IUPAC. Pure and App. Chem. Vol 67, nº 4, 649-666, 1995
• "Shewhart Control Charts" ISO 8258. 1991.
• "Precision of test methods - Determination of repeatability and reproducibility for a standard test method by inter-laboratory tests". ISO 5725, 1994.
• "Draft Commission Regulation of establishing rules for the application of reference and routine methods for the analysis and quality evaluation of milk and milk products". Commission of the European Communities, 1995.
• "Harmonized protocols for the adoption of standardized analytical methods and for the presentation of their performance characteristics". IUPAC. Pure an App. Chem., Vol. 62, nº 1, 149-162. 1990.

OIV-MA-AS1-09 Protocol for the designe, conducts and interpretation of collaborative studies

Introduction

After a number of meetings and workshops, a group of representatives from 27 organizations adopted by consensus a "Protocol for the design, conducts and interpretation of collaborative studies" which was published in Pure & Appl. Chem. 60, 855-864, 1995. A number of organizations have accepted and used this protocol. As a result of their experience and the recommendations of the Codex Committee on Methods of Analysis and Sampling (Joint FAO/WHO Food Standards Programme, Report of the Eighteenth Session, 9-13 November, 1992; FAO, Rome Italy, ALINORM 93/23, Sections 34-39), three minor revisions were recommended for incorporation into the original protocol. These are: (1) Delete the double split level design because the interaction term it generates depends upon the choice of levels and if it is statistically significant, the interaction cannot be physically interpreted. (2) Amplify the definition of "material". (3) Change the outlier removal criterion from 1% to 2.5%.

The revised protocol incorporating the changes is reproduced below. Some minor editorial revisions to improve readability have also been made. The vocabulary and definitions of the document 'Nomenclature of Interlaboratory Studies (Recommendations 1994)' [published in Pure Appl Chem., 66, 1903-1911 (1994)] has been incorporated into this revision, as well as utilizing, as far as possible, the appropriate terms of the International Organization for Standardization (ISO), modified to be applicable to analytical chemistry.

Protocol

1. Preliminary work

Method-performance (collaborative) studies require considerable effort and should be conducted only on methods that have received adequate prior testing. Such within-laboratory testing should include, as applicable, information on the following:

1.1.  Preliminary estimates of precision

Estimates of the total within-laboratory standard deviation of the analytical results over the concentration range of interest as a minimum at the upper and lower limits

of the concentration range, with particular emphasis on any standard or specification value.

Note 1: The total within-laboratory standard deviation is a more inclusive measure of imprecision that the ISO repeatability standard deviation, §3.3 below. This standard deviation is the largest of the within-laboratory type precision variables to be expected from the performance of a method; it includes at least variability from different days and preferably from different calibration curves. It includes between-run (between-batch) as well as within-run (within-batch) variations. In this respect it can be considered as a measure of within-laboratory reproducibility. Unless this value is well within acceptable limits, it cannot be expected that the between-laboratory standard deviation (reproducibility standard deviation) will be any better. This precision term is not estimated from the minimum study described in this protocol.

NOTE 2: The total within-laboratory standard deviation may also be estimated from ruggedness trials that indicate how tightly controlled the experimental factors must be and what their permissible ranges are. These experimentally determined ranges should be incorporated into the description of the method.

1.2.  Systematic error (bias)

Estimates of the systematic error of the analytical results over the concentration range and in the substances of interest, as a minimum at the upper and lower limits of the concentration range, with particular emphasis on any standard or specification value.

The results obtained by applying the method to relevant reference materials should be noted.

1.3.  Recoveries

The recoveries of "spikes" added to real materials and to extracts, digests, or other treated solutions thereof.

1.4.  Applicability

The ability of the method to identify and measure the physical and chemical forms of the analyte likely to be present in the materials, with due regard to matrix effects.

1.5.  Interference

The effect of other constituents that are likely to be present at appreciable concentrations in matrices of interest and which may interfere in the determination.

1.6.  Method comparison

The results of comparison of the application of the method with existing tested methods intended for similar purposes.

1.7.  Calibration Procedures

The procedures specified for calibration and for blank correction must not introduce important bias into the results.

1.8.  Method description

The method must be clearly and unambiguously written.

1.9.  Significant figures

The initiating laboratory should indicate the number of significant figures to be reported, based on the output of the measuring instrument.

Note: In making statistical calculations from the reported data, the full power of the calculator or computer is to be used with no rounding or truncating until the final reported mean and standard deviations are achieved. At this point the standard deviations are rounded to 2 significant figures and the means and related standard deviations are rounded to accommodate the significant figures of the standard deviation. For example, if = 0.012, c is reported as 0.147, not as 0. 1473 or 0. 15, and RSD_R is reported as 8.2%. (Symbols are defined in Appendix L) If standard deviation calculations must be conducted manually in steps, with the transfer of intermediate results, the number of significant figures to be retained for squared numbers should be at least 2 times the number of figures in the data plus 1.

2. Design of the method-performance study

2.1.  Number of materials

For a single type of substance, at least 5 materials (test samples) must be used; only when a single level specification is involved for a single matrix may this minimum required number of materials to be reduced to 3. For this design parameter, the two portions of a split level and the two individual portions of blind replicates per laboratory are considered as a single material.

Note 1: A material is an 'analyte/matrix/concentration' combination to which the method-performance parameters apply. This parameter determines the applicability of a method. For application to a number of different substances, a sufficient number of matrices and levels should be chosen to include potential interferences and the concentration of typical use.

Note 2: The 2 or more test samples of blind or open replicates statistically, are a single material (they are not independent).

NOTE 3: A single split level (Youden pair) statistically analyzed as a pair is a single material; if analyzed statistically and reported as single test samples, they are 2 materials. In addition, the pair can be used to calculate the within-laboratory standard deviation, as

where , the difference between the 2 individual values from the split level for each laboratory and n is the number of laboratories. In this special case, , the among laboratories standard deviation, is merely the average of the two values calculated from the individual components of the split level, and it is used only as a check of the calculations.

Note 4: The blank or negative control may be a material or not depending on the usual purpose of the analysis. For example, in trace analysis, where very low levels (near the limit of quantitation) are often sought, the blanks are considered as materials and are necessary to determine certain 'limits of measurement.' However, if the blank is merely a procedural control in macro analysis (e.g., fat in cheese), it would not be considered a material.

2.2.  Number of laboratories

At least 8 laboratories must report results for each material; only when it is impossible to obtain this number (e.g., very expensive instrumentation or specialized laboratories required) may the study be conducted with less, but with an absolute minimum of 5 laboratories. If the study is intended for international use, laboratories from different countries should participate. In the case of methods requiring the use of specialized instruments, the study might include the entire population of available laboratories. In such cases, "n" is used in the denominator for calculating the standard deviation instead of "(n - 1)". Subsequent entrants to the field should demonstrate the ability to perform as well as the original participant.

2.3.  Number of Replicates

The repeatability precision parameters must be estimated by using one of the following sets of designs (listed in approximate order of desirability):

2.3.1.      Split Level

For each level that is split and which constitutes only a single material for purposes of design and statistical analysis, use 2 nearly identical test samples that differ only slightly in analyte concentration (e.g., <1-5%). Each laboratory must analyse each test sample once and only once.

Note: The statistical criterion that must be met for a pair of test samples to constitute a split level is that the reproducibility standard deviation of the two parts of the single split level must be equal.

2.3.2.      Combination blind replicates and split level

Use split levels for some materials and blind replicates for other materials in the same study (single values from each submitted test sample).

2.3.3.      Blind replicates

For each material, use blind identical replicates, when data censoring is impossible (e.g., automatic input, calculation, and printout) non-blind identical replicates may be used.

2.3.4.      Known replicates

For each material, use known replicates (2 or more analyses of test portions from the same test sample), but only when it is not practical to use one of the preceding designs.

2.3.5.      Independent analyses

Use only a single test portion from each material (i.e., do not perform multiple analyses) in the study, but rectify the inability to calculate repeatability parameters by quality control parameters or other within-laboratory data obtained independently of the method-performance study.

3. Statistical analysis (See Flowchart, A.4. 1)

For the statistical analysis of the data, the required statistical procedures listed below must be performed and the results reported. Supplemental, additional procedures are not precluded.

3.1.  Valid data

Only valid data should be reported and subjected to statistical treatment. Valid data are those data that would be reported as resulting from the normal performance of laboratory analyses; they are not marred by method deviations, instrument malfunctions, unexpected occurrences during performance, or by clerical, typographical and arithmetical errors.

3.2.  One-way analysis of variance

One-way analysis of variance and outlier treatments must be applied separately to each material (test sample) to estimate the components of variance and repeatability and reproducibility parameters.

3.3.  Initial estimation

Calculate the mean, c (= the average of laboratory averages), repeatability relative standard deviation, and reproducibility relative standard deviation, RSD_R with no outliers removed, but using only valid data.

3.4.  Outlier treatment

The estimated precision parameters that must also be reported are based on the initial valid data purged of all outliers flagged by the harmonized 1994 outlier removal procedure. This procedure essentially consists of sequential application of the Cochran and Grubbs tests (at 2.5% probability (P) level, 1-tail for Cochran and 2-tail for Grubbs) until no further outliers are flagged or until a drop of 22.2% (= 219) in the original number of laboratories providing valid data would occur.

Note: Prompt consultation with a laboratory reporting suspect values may result in correction of mistakes or discovering conditions that lead to invalid data, 3.1.

Recognizing mistakes and invalid data per se is much preferred to relying upon statistical tests to remove deviate values.

3.4.1.      Cochran test

First apply Cochran outlier test (1-tail test a P = 2.5%) and remove any laboratory whose critical value exceeds the tabular value given in the tale, Appendix A.3. 1, for the number of laboratories and replicates involved.

3.4.2.      Grubbs tests

Apply the single value Grubbs test (2 tail) and remove any outlying laboratory. If no laboratory is flagged, then apply the pair value tests (2 tail) - 2 at the same end and 1 value at each end, P = 2.5% overall. Remove any laboratory(ies) flagged by these tests whose critical value exceeds the tabular value given in the appropriate column of the table Appendix A.3.3. Stop removal when the next application of the test will flag as table, A outliers more that 22.2% (2 of 9) of the laboratories.

Note: The Grubbs tests are to be applied one material at a time to the set of replicate means from all laboratories, and not to the individual values from replicated designs because the distribution of all the values taken together is multimodal, not Caussian, i.e., their differences from the overall mean for that material are not independent.

3.4.3.      Final estimation

Recalculate the parameters as in §3.3 after the laboratories flagged by the preceding procedure have been removed. If no outliers were removed by the Cochran-Grubbs sequence, terminate testing. Otherwise, reapply the Cochran-Grubbs sequence to the data purged of the flagged outliers until no further outliers are flagged or until more than a total of 22.2% (2 of 9 laboratories) would be removed in the next cycle. See flowchart A.3.4.

4. Final report

The final report should be published and should include all valid data. Other information and parameters should be reported in a format similar (with respect to the reported items) to the following, as applicable:

[x] Method-performance tests carried out at the international level in [year(s)] by [organisation] in which [y and z] laboratories participated, each performing [k] replicates, gave the following statistical results:

Table of method -Performance parameters

Analyte; Results expressed in [units]

Material [Description and listed in columns across top of table in increasing order of magnitude of means]

Number of laboratories retained after eliminating outliers

Number of outlying laboratories

Code (or designation) of outlying laboratories

Number of accepted results

Mean

True or accepted value, if known

Repeatability standard deviation (S_r)

Repeatability relative standard deviation (RSD_R)

Repeatability limit, r (2.8 x S_r)

Reproducibility standard deviation (S_R)

Reproducibility relative standard deviation (RSD_R)

Reproducibility limit, R (2.8 X S_R)

4.1.  Symbols

A set of symbols for use in reports and publications is attached as Appendix 1 (A.1.).

4.2.  Definitions

A set of definitions for use in study reports and publications is attached as Appendix 2 (A.2.).

4.3.  Miscellaneous

4.3.1.      Recovery

Recovery of added analyte as a control on method or laboratory bias should be calculated as follows:

[Marginal] Recovery, %=

(Total analyte found - analyte originally present) x 100/(analyte added)

Although the analyte may be expressed as either concentration or amount, the units must be the same throughout. When the quantity of analyte is determined by analysis, it must be determined in the same way throughout.

Analytical results should be reported uncorrected for recovery. Report recoveries separately.

4.3.2.      When , is negative

By definition, is greater than or equal to   in method-performance studies; occasionally the estimate of is greater than the estimate of (the average of the replicates is greater than the range of laboratory averages and the calculated is then negative). When this occurs, set = 0 and = .

5. References

• Horwitz, W. (1988) Protocol for the design, conduct, and interpretation of method performance studies. Pure & Appl. Chem. 60, 855-864.
• Pocklington, W.D. (1990) Harmonized protocol for the adoption of standardized analytical methods and for the presentation of their performance characteristics. Pure and Appl. Chem. 62, 149-162.
• International Organization for Standardization. International Standard 5725-1986. Under revision in 6 parts; individual parts may be available from National Standards member bodies.

Appendices

Appendix 1. - Symbols

Use the following set of symbols and terms for designating parameters developed by a method-performance study.

Mean (of laboratory averages): x

Standard deviations:s (estimates)

• Repeatability:
• 'Pure' between-laboratory:
• Reproducibility;

Variances: (with subscripts, r, L, and R)

Relative standard deviations: RSD (with subscripts, r, L, and r)

Maximum tolerable differences

(as defined by ISO 5725-1986);

See A.2.4 and A.2.5)

Repeatability limitr = (2.8 x )

Reproducibility limit R = (2.8 X )

Number of replicates per laboratory :k (general)

Average number of replicates per laboratory i:k (for a balanced design)

Number of laboratories :L

Number of materials (test samples): m

Total number of values in a given assay: n (= kL for a balanced design)

Total number of values in a given study: N (= kLm for an overall balanced design)

____________________

If other symbols are used, their relationship to the recommended symbols should be explained fully.

Appendix 2. -  Definitions

Use the following definitions. The first three definitions utilize the 1UPAC document "Nomenclature of Interlaboratory Studies" (approved for publication 1994). The next two definitions are assembled from components given in ISO 3534-1:1993. All test results are assumed to be independent, i.e., 'obtained in a manner not influenced by any previous result on the same or similar test object. Quantitative measures of precision depend critically on the stipulated conditions. Repeatability and reproducibility conditions are particular sets of extreme stipulated conditions.'

1. A.2.1 Method-performance studies

An interlaboratory study in which all laboratories follow the same written protocol and use the same test method to measure a quantity in sets of identical test items [test samples, materials]. The reported results are used to estimate the performance characteristics of the method. Usually these characteristics are within-laboratory and among-laboratories precision, and when necessary and possible, other pertinent characteristics such as systematic error, recovery, internal quality control parameters, sensitivity, limit of determination, and applicability.

2. A.2.2 Laboratory-performance study

An interlaboratory study that consists of one or more analyses or measurements by a group of laboratories on one or more homogeneous, stable test items, by the method selected or used by each laboratory. The reported results are compared with those of other laboratories or with the known or assigned reference value, usually with the objective of evaluating or improving laboratory performance.

3. A.2.3 Material certification stud

An interlaboratory study that assigns a reference value ('true value') to a quantity (concentration or property) in the test item, usually with a stated uncertainty.

4. A.2.4  Repeatability limit (r)

When the mean of the values obtained from two single determinations with the same method on identical test items in the same laboratory by the same operator using the same equipment within short intervals of time, lies within the range of the mean values cited in the Final Report, 4.0, the absolute difference between the two test results obtained should be less than or equal to the repeatability limit (r) [= 2.8 x s,) that can generally be inferred by linear interpolation of from the Report.

Note: This definition, and the corresponding definition for reproducibility limit, has been assembled from five cascading terms and expanded to permit application by interpolation to a test item whose mean is not the same as that used to establish the original parameters, which is the usual case in applying these definitions. The term 'repeatability [and reproducibility] limit' is applied specifically to a probability of 95% and is taken as 2.8 x s, [or SRI. The general term for this statistical concept applied to any measure of location (e.g., median) and with other probabilities (e.g., 99%) is "repeatability [and reproducibility] critical difference".

5. A.2.5  Reproducibility limit (R)

When the mean of the values obtained from two single determinations with the same method on identical test items in different laboratories with different operators using different equipment, lies within the range of the mean values cited in the Final Report, 4.0, the absolute difference between the two test results obtained should be less than or equal to the reproducibility limit (R) [= 2.8 x ] that can generally be inferred by linear interpolation of from the Report.

Note 1: When the results of the interlaboratory test make it possible, the value of r and R can be indicated as a relative value (e.g., as a percentage of the determined mean value) as an alternative to the absolute value.

Note 2: When the final reported result in the study is an average derived from more than a single value, i.e., k is greater than 1, the value for R must be adjusted according to the following formula before using R to compare the results of a single routine analyses between two laboratories.

Similar adjustments must be made for replicate results constituting the final values for and , if these will be the reported parameters used for quality control purposes.

Note 3: The repeatability limit, r, may be interpreted as the amount within which two determinations should agree with each other within a laboratory 95% of the time. The reproducibility limit, R, may be interpreted as the amount within which two separate determinations conducted in different laboratories should agree with each other 95% of the time.

Note 4: Estimates Of can be obtained only from a planned, organized method performance study; estimates of can be obtained from routine work within a laboratory by use of control charts. For occasional analyses, in the absence of control charts, within-laboratory precision may be approximated as one half S_R (Pure and Appl. Chem., 62, 149-162 (1990) , Sec. L3, Note.).

6. A.2.6 One-way analysis of variance

One-way analysis of variance is the statistical procedure for obtaining the estimates of within laboratory and between-laboratory variability on a material-by-material basis. Examples of the calculations for the single level and single-split-level designs can be found in ISO 5725-1986.

Appendix 3. – Critical values

1. A.3.1 Critical values for the Cochran maximum variance ratio at the 2.5% (1 -tail) rejection level, expressed as the percentage the highest variance is of the total variance; r = number of replicates.

Tables A.3.1 and A.3.3 were calculated by R. Albert (October, 1993) by computer simulation involving several runs of approximately 7000 cycles each for each value, and then smoothed. Although Table A.3.1 is strictly applicable only to a balanced design (same number of replicates from all laboratories), it can be applied to an unbalanced design without too much error, if there are only a few deviations.

2. A.3.2 Calculation of Cochran maximum variance outlier ratio

Compute the within-laboratory variance for each laboratory and divide the largest of these variances by the sum of the all of the variances and multiply by 100. The resulting quotient is the Cochran statistic which indicates the presence of a removable outlier if this quotient exceed the critical value listed above in the Cochran table for the number of replicates and laboratories specified.

3. A.3.3 Critical values for the Grubbs extreme deviation outlier tests at the 2.5% (2-tail), 1.25% (1tail) rejection level, expressed as the percent reduction in standard deviations caused by the removal of the suspect value(s).

4. A.3.4 Calculation of the Grubbs test values

To calculate the single Chubbs test statistic, compute the average for each laboratory and then calculate the standard deviation (M) of these L averages (designate as the original s). Calculate the SD of the set of averages with the highest average removed (SH); calculate the SD of the set of averages with the lowest average removed (SL). The calculate the percentage decrease in SD for both as follows:

• 100 x [ 1 - (sL/s] and 100 x [ 1 - (sH/s)].

The higher of these two percentage decreases is the singe Grubbs test statistic, which signal the presence of an outlier to be omitted at the P = 2.5% level, 2tail, if it exceeds the critical value listed in the single value column, Column 2, of Table A.3.3 , for the number of laboratory averages used to calculate the original s.

To calculate the paired Grubbs test statistics, calculate the percentage decrease in standard deviation obtained by dropping the two highest averages and also by dropping the two lowest averages, as above. Compare the higher of the percentage changes in standard deviation with the tabular values in column 3 and proceed with (1) or (2): (1) If the tabular value is exceeded, remove the responsible pair. Repeat the cycle again, starting at the beginning with the Cochran extreme variance test again, the Grubbs extreme value test, and the paired Grubbs extreme value test. (2) If no further values are removed, then calculate the percentage change in standard deviation obtained by dropping both the highest extreme value and the lowest extreme value together, and compare with the tabular values in the last column of A.3.3. If the tabular value is exceeded, remove the high-low pair of averages, and start the cycle again with the Cochran test until no further values are removed. In all cases, stop outlier testing when more than 22.2% (2/9) of the averages are removed.

Appendix 4

1. A.4.1. Flowchart for outlier removal

OIV-MA-AS1-10 Estimation of the detection and quantification limits of a method of analysis

1. Purpose: to establish the detection and quantification limits of a method

N.B. : The proposed calculation procedure sets « detection and quantification limiting » values with respect to the instrumental response. For a given method, the final calculation of these values must take cognizance of factors arising from the preparation of the sample.

2. Definitions

Detection limit: the smallest concentration or proportion of the analyzed substance that can be detected with an acceptable level of uncertainty, but that is not quantified under the experimental conditions described in the method

Quantification limit: the smallest concentration or proportion of the analyzed substance  that can be quantified with an acceptable level of uncertainty, under the experimental conditions described in the method.

3. Logic Diagram for Decision-Making

4. Methodology

4.1.  "Results" approach

colorimetry), the detection limit (L_D) and the quantification limit (L_Q) are estimated using one of the two following methods.

4.1.1.      Method 1:

Directly read n measurements (analyte quantity or response) of separate analytic  « blank » samples that contain all of the constituents, with the exception of  the substance to be tested for.

where and are the mean and standard deviation for n measurements.

Note: A multiplication factor of 3 corresponds to a 0.13% chance of concluding that the substance sought is present, when, in fact, it is lacking.  10 corresponds to a 0.5% chance.

4.1.2.      Method 2:

Using the straight calibration line: Y = a + bX

The detection limit is the smallest concentration of a substance that can be distinguished from the blank, with a 0.13% risk of retaining samples containing nothing ; in other words, the value  beginning at which a statistical test comparing the response to  0 becomes significant with an error level α of 0.13%.  Hence:

Where Sa is the standard deviation on the ordinate at the origin of the straight regression line.  The logic is the same for L_Q,, where the multiplication factor is 10 (risk of 0.5%).

4.2.  "Graph" Approach

For analytical methods which generate graphs (i.e., chromatography), the detection limit is estimated based on the ground noise of the analytic blank recording for a given sample.

= 3 x h x R (associated risk is below 0.13%) and

= 10 x h x R (associated risk is below 0.5%), where

h is the average or maximum amplitude of the signal window corresponding to 10 width s of the mid-height peak on either side of the retention time, as a function of stability.

R is the quantity/signal response factor expressed as a function of  the quantity of substance/height.

On each occasion, three series of three injections each are performed on test blanks at an interval of several days.

4.2.1.      method

Increase ground noise to the maximum (Fig. 1 above) ;

center around the retention time (RT) of the product ;

draw a window of 10 widths of the mid-height peak (W1/2) on either side of the RT ;

draw two parallel lines, one running through the highest point of the highest peak, the other through the base of the deepest trough ;

evaluate height -> h_max ;

calculate the response factor (R factor) ;

= 3 x x R

= 10 x x R

4.2.2.      Method

increase the ground noise to the maximum (Fig. 2 above);

center around the retention time (RT) of the product;

draw a window of 10 widths of the mid-height peck (W1/2) on either side of the RT;

divide into 20 equal sections (x);

draw two parallel lines in each block, one running through the highest point of  the highest peak, the other through the base of the deepest trough ;

measure the heights, y;

calculate the average (y = );

calculate the response factor (R factor);

= 3 x x R;

= 10 x x R

These estimates can themselves be validated by injecting quantities of solute  that are close to the calculated limits (Figures 3 and 4).

Compound at [c]  #

Figure No. 3: Validating calculations of limits.

Concentration of the compound approaches

N.B. : The dotted line corresponds to the real injected value  however, since this figure is provided as an example, it may be deleted from the final text

Compound at < [c] <

Figure No. 4: Validating calculations of limits.

Concentration of compound between and

N.B. : The dotted line corresponds to the real injected value; however, since this figure is provided as an example, it may be deleted from the final text.