Dynamic, Rule-based Quality Control Framework for Real-time Sensor Data
Wade M. Sheldon
Last modified: 2008-08-21
The volume of monitoring data acquired and managed by Long Term Ecological Research Sites and environmental observatories has increased exponentially over time, thanks to advances in sensor technology and computing power combined with steady decreases in data storage costs. New directions in environmental monitoring, such as high density sensor networks and autonomous roving sensors, promise to raise the bar even higher. Quality control is often a major challenge with real-time data streams, though, due to poor scalability of traditional software tools, approaches and analysis methods. Software developed at the Georgia Coastal Ecosystems Long Term Ecological Research Site (GCE Data Toolbox for MATLAB) has proven very effective for quality control of real-time data, as well as interactive analysis during post processing and synthesis. This paper describes the dynamic, rule-based quality control framework provided by this software, and presents quantitative performance data that demonstrate these tools can efficiently perform quality analysis on million-record data sets using commodity computer hardware.