Purpose
Data need to be complete, usable, and actionable. To achieve this, NSSP's BioSense Platform checks core data elements in records before processing. A useful analogy is to compare the data flow to baseball. These tips will help users understand how to round the bases and keep records out of the filtered zone.
Introduction
Records being processed into NSSP’s BioSense Platform are checked for core data elements before being passed to the next stage of data processing. Like in baseball, it begins at home base. For NSSP, home base is when data arrive on the BioSense Platform. Arrival—or, stepping up to the plate—isn’t enough. These data must be ready to perform to be passed to first base, the Raw Table.
Aim for first base
Each record is checked to make sure its contents meet message standards. Messages with all the essential information get a status of "Read" and are ready for processing (rounding the bases). If minimal standards are not met, the record is filtered and stored in the Raw Table with a status of "Filtered" (message_status='Filtered'). Filtered records are not processed.
Records are filtered for three main reasons:
1. Message lacks date/time from MSH_7.1 segment
2. MSH_9.1 Not Equal to ADT
3. MSH-4 lacks sending facility information
So, how can you keep your records out of the filtered zone? You do this by making sure each facility and vendor send essential data elements. Check your monthly Data Quality Reports to see how many records are being filtered. Then, as needed, contact the associated facilities or vendors for further action.
Rounding the bases
Once the record passes the initial filtering test and makes it to the Raw Table (first base), another stringent test must be passed to reach the Processed Table—or, second base. This check makes sure core data elements are present. And if not, records are removed and placed in a separate, site-specific Exceptions Table. Once the record passes the Exceptions check, the record is prepped per business processing rules and collapsed into holistic, visit-level data. Then the record is ready to round second base (Processing) toward third base, which is ESSENCE.
After ESSENCE processing, site personnel will be able to visualize these data and conduct additional analyses. Data from the Raw, Exceptions, and Processed Tables then head to Home Base and into the DataMart, where site personnel can use tools such as SAS Studio or Posit Workbench to further access these data.
To recap: Records can be Filtered to the Raw Table, sent to the Exceptions Table, or successfully passed to the Processed Table and into ESSENCE. The Processed Table and ESSENCE are the end goals to ensure you have complete, usable, and actionable data. Filtered and exceptions Tables are the places where records can be stopped before getting there.
With your help in maintaining and improving data quality, more and more data are rounding the bases and hitting home runs! We hope you find this explanation helpful. Please email nssp@cdc.gov if you have questions.