10.4 ASSESSING THE QUALITY ASSURANCE SYSTEM FOR THE OKLAHOMA MESONET WITH ACCURACY MEASURES Peter K. Hall Jr., Alexandria G. McCombs, Christopher A. Fiebrich, and Renee A. McPherson* Oklahoma Mesonet, Norman, Oklahoma 1. INTRODUCTION Quality assurance (QA) meteorologists for the Oklahoma Mesonet issue a “trouble ticket” when they detect a problem with a particular sensor (McPherson et al. 2007). The trouble ticket indicates to a Mesonet technician that a repair or replacement is needed for a specific sensor at a specific Mesonet site. Sometimes the resolution of the sensor problem simply requires rewiring or adjustment; other times the sensor needs to be replaced. Sensors also can be replaced on a pre- assigned basis (i.e., scheduled rotation). The “rotated” sensors are not changed because of a particular problem, but when they have been in the field for a predetermined, sensor- dependent length of time (Fiebrich et al. 2006). When sensors are replaced, either because of a problem or rotation, the old sensors are returned to the calibration laboratory for an “as found” test. The “as found” test compares the sensor to a reference sensor of similar type. At this point, a lab technician can diagnose if the sensor needs to be reconditioned, repaired, or retired, or if there was no problem. Clearly, the Mesonet QA staff strives to minimize the number of trouble tickets issued on sensors that do not have problems. The “as found” tests performed on the sensors returning from the field (regardless of why a trouble ticket was issued) are important to the QA staff to determine if a problem was identified correctly or if a rotated sensor had a sensor problem. Accuracy measures and skill scores typically have been applied to forecast verification (e.g., Mason 1982; Doswell et al. 1990), but were used here to assess the quality assurance process completed by manual methods. This paper focuses on accuracy measures of seven different variables, as calculated from the “as found” sensor tests of the Oklahoma Mesonet. 2. ACCURACY MEASURE PROCEDURE Accuracy measures were calculated on the following variables air temperature at 1.5 m (TAIR) from a fast-response thermistor (hereafter “fasttherm”), air temperature at 1.5 m (TSLO) from a slower-response thermistor, relative humidity at 1.5 m (RELH), soil temperatures at 5, 10, and 30 cm (SOIL), pressure (PRES), wind speed at 2 or 9 m (WSEN), and wind speed at 10 m (WSPD). The accuracy measures calculated for these seven variables were based on a dichotomous contingency table (Table 1; Wilks 1995). If a trouble ticket were issued and the sensor failed calibration, a “Hit” occurred. However, if a ticket were issued but the sensor did not fail calibration, a “False Alarm” was counted. A “Miss” constituted a sensor that returned to the lab for rotation but it failed the calibration test (i.e., the QA staff did not detect a confirmed problem). Finally, a “Correct Negative” referred to a sensor that returned for rotation and did not fail the calibration test. Calibration sheets for calendar year 2007 were analyzed to place the results of a given sensor test in one of the four categories. The sum of each category was used to calculate specific accuracy measures, as detailed in equations 1–5. The total number of events (“Total Number”) also was obtained. ––––––––––––––––––––––––––––––––––––––– * Corresponding author address: Renee A. McPherson, Oklahoma Climatological Survey, University of Oklahoma, 120 David L. Boren Blvd., Suite 2900, Norman, OK 73072– 7305; email: renee@ou.edu .