Accuracy and Precision

Important considerations

Accuracy and precision are important considerations when radiographic classifications made.

Accuracy - the ability for a measurement to reflect a true degree of underlying abnormality.

Precision - the extent a measurement is consistent across repeated determinations.

A measurement technique can be precise but inaccurate or can be accurate and imprecise. It is preferable for a measurement to be both accurate and precise to optimize validity.

Selecting readers

To achieve the best accuracy, readers should be both proficient and experienced in classifying chest radiographs for pneumoconioses. Ideally, readers should be:

• Current B Readers
• Highly experienced in classifying radiographs of dust-exposed workers
• Representative of general classification practices among readers
  • Not falling at either end of the extremes of the range of inter-reader variability

Reader selection procedures that give rise to unbiased classifications include:

• Selection based on pre-existing evidence of mainstream classification tendencies
• Random selection from the largest a pool of available readers

Selecting readers based on other criteria leaves the process open to accusations of bias. Proper procedures for selecting readers are not alone sufficient to ensure accuracy. Procedures should be accompanied by an appropriate quality assurance program.

Multiple readers and precision

Rigorous training is preferred to ensure accuracy before evaluating a reader. Use a pilot reading trial and quality control (calibration) readings of radiographs with known classifications. Keep in mind, despite careful training, evaluation, and feedback, systematic reader differences can continue.

Multiple readings with appropriate summary measures (e.g., the median reading) can minimize the impact of any one reader on final determinations. Multiple readings also help improve precision of the data.

Precision is achieved by using summary (e.g. median) scores derived from multiple independent classifications by different readers who classify images independently. Independent classification means classification without other readers being present and without knowledge of other readers' classifications. The number of independent classifications obtained depends on the setting and monetary costs involved.

Summary classifications developed from independent classifications are more precise than single individual classifications. However, care should be taken not to introduce bias when deriving summary classifications. Valid summarization methods include using median classifications or properly designed consensus measures.

Using reader panels where groups of readers jointly classify radiographs and come to a consensus or unanimous decision is not recommended. Apart from logistical difficulties of convening such panels, decisions made may fail to represent the true range of the group's opinions. Instead, joint classifications may reflect those of the most dominant or experienced reader or readers in the group.

Classification binding

Overall bias can occur when readers have information about radiographs being classified that can consciously or unconsciously influence their classifications. Knowing about worker exposures can bias readers to recording more or fewer abnormalities depending on the extent of an exposure. It can also cause preferential selection of certain types of abnormality depending on the nature of the exposure. For example, small, rounded opacities for silica-exposed workers versus small irregular opacities for asbestos-exposed workers.

Blinding readers allows a classification to be made absent of preconceived knowledge and concepts. To minimize bias, remove or obscure identifying information before sending radiographs. Identifying information includes:

• Age
• Occupation
• Work site information
• Medical history

Withholding information on the source of the radiographs will prevent bias. Additionally, withholding who the classification has been requested for (e.g., the plaintiff or defendant in contested proceedings) will also prevent bias.

Blinding is not appropriate for medical diagnosis and worker medical monitoring. However, if radiographs from worker monitoring programs are used for epidemiologic studies, accuracy and reproducibility of results may be improved. This can be achieved by having radiographs re-read blinded to information that might influence a reader's classification. Examples of such information include industry, occupation, tenure, etc.

Knowing the order radiographs were taken influences reader classification when assessing epidemiologic temporal trends in disease development. It also influences progression using sequential radiographs1.

Extreme determinations

Rewarding a reader for reporting disease leads to bias. Payment or compensation should not be linked with the outcome reported by the reader. Those seeking accurate classifications should not select readers whose classifications are likely biased in a direction that suits their preference.2