Validation Process Enhances Annual Publication Data Quality

Overview

As part of CDC's Data Modernization Initiative, the National Notifiable Disease Surveillance System (NNDSS) redesigned how health departments in 60 state, local, and territorial jurisdictions send data about cases of over 120 nationally notifiable diseases and conditions to CDC. Through NNDSS, CDC receives and uses these data to keep people healthy and defend America from health threats.

Public health problem

CDC publishes NNDSS' Annual Tables through CDC Wide-ranging Online Data for Epidemiologic Research (CDC WONDER). The annual tables are a key data source for public health action. They also highlight cases of selected infectious national notifiable diseases from NNDSS data. Health departments in the 50 states, New York City, the District of Columbia, and the U.S. territories report these cases to CDC. Because they represent final and official data for each year, CDC must ensure the highest accuracy for these data before publishing.

For over three decades, NNDSS used statistical analysis software and spreadsheets to create and manage annual datasets. The old process required CDC staff to manually run separate data processing programs at several points in the data pipeline. Manual steps made the data pipeline cumbersome, time consuming, and vulnerable to human error.

Actions taken

In fall 2022, the NNDSS team:

• Transitioned away from the decades-old process.
• Implemented improvements to the processing and publication of weekly and annual case count tables for over 120 national notifiable diseases and conditions.

With the new fully automated process and enhancements, CDC is now better positioned to validate the data. CDC can now confirm the information in the NNDSS database matches information in jurisdiction's database before finalizing data for the year.

This major accomplishment builds on a comprehensive strategy to enhance the Message Validation, Processing, and Provisioning System (MVPS) and streamline and automate case data processing and publication.

In addition to developing, testing, and implementing technology solutions, the team coordinated closely with CDC disease programs to develop clear, concise documentation and business rules for the new publication process. These resources ensure that everyone involved in annual data publication has a shared understanding of roles and responsibilities.

Core NNDSS team member Roseanne English describes the complexity of this effort. "Our team built many new processes to validate data at every step. We had to change our business requirements into the right technical solutions so they could be more easily understood by MVPS," says English. "We had to ensure nothing was lost in translation as development teams entered the design and implementation phase," said English. In addition, we designed and implemented robust and thorough data validation processes to verify the accuracy of data across all the annual tables, for all combinations of jurisdictions and national notifiable data represented in the tables."

Impact

The team is strengthening and unifying critical health data across the nation to better serve the public by:

• Developing new processes.
• Modernizing existing standards.
• Transitioning to a more sustainable data-processing system.

The team can now publish the annual tables faster and more efficiently, with fewer data processing errors.

With this redesign, CDC is poised to speed up data reconciliation efforts and increase capacity for future public health responses. This investment in modernizing the processing and validation of data to support NNDSS publications can also be leveraged year to year to support future annual publications.

To learn more about how NNDSS is partnering across CDC, state, local, and territorial health departments to better understand, track, and respond to health threats, visit the Case Surveillance in Action section of the NNDSS website.