Journal Information
Journal ID (publisher-id): OJPHI
ISSN: 1947-2579
Publisher: University of Illinois at Chicago Library
Article Information
©2013 the author(s)
open-access: This is an Open Access article. Authors own copyright of their articles appearing in the Online Journal of Public Health Informatics. Readers may copy articles without permission of the copyright owner(s), as long as the author and OJPHI are acknowledged in the copy and the copy is used for educational, not-for-profit purposes.
Electronic publication date: Day: 4 Month: 4 Year: 2013
collection publication date: Year: 2013
Volume: 5E-location ID: e38
Publisher Id: ojphi-05-38

Automated Surveillance of Outpatients with Pneumonia: A Performance Evaluation
Hongzhang Zheng12
Tariq Siddiqui12
Sylvain DeLisle*12
1VA Maryland Health Care System, Baltimore, MD, USA;
2Medicine, University of Maryland, Baltimore, MD, USA
*Sylvain DeLisle, E-mail:


To determine if influenza surveillance should target all patients with acute respiratory infections (ARI) or only track pneumonia cases.


Effective responses to epidemics of infectious diseases hinge not only on early outbreak detection, but also on an assessment of disease severity. In recent work, we combined previously developed ARI case-detection algorithms (CDA) [1] with text analyses of chest imaging reports to identify ARI patients whose providers thought had pneumonia. In this work, we asked if a surveillance system aimed at patients with pneumonia would outperform one that monitors the full severity spectrum of ARI.


Time series of daily casecounts (backgrounds) were created by applying either an ARI CDA (ARI ICD-9 codeset [1]) or a Pneumonia CDA (ARI ICD-9 codes AND chest imaging obtained AND positive results from automated text analysis that identify those chest imaging reports that support the diagnosis of pneumonia) to electronic medical record (EMR) entries related to outpatient encounters at the VA Maryland Health Care System. We used an age-structured metapopulation influenza epidemic model for Baltimore to inject factitious influenza cases into backgrounds. Injections were discounted by the known sensitivity of the ARI CDA [1]. For injections into the pneumonia backgrounds time series, factitious ARI cases were further discounted by the expected pneumonia rate in the modeled influenza epidemic (10%). From the time of injection, EARS or CUSUM statistics [2,3] were applied on each successive day on paired back-ground+injection vs. background-only time series. Each injection-prospective-surveillance cycle was repeated 52 times, each time with the injection shifted to a different week of the one-year study period (2010–11). We computed: 1) the “Detection Delay”, the average time from injection to the first alarm present in the back-ground+injection dataset but absent from the background-only dataset; 2) the “False-Alarm Rate” (FAR), defined as the number of unique false-alarms originating in the background-only dataset during the study year, divided by 365 days. To create activity monitoring operating characteristic (AMOC) curves, we empirically determined the corresponding Delay-FAR pairs over a wide range of alarm thresholds.


The Figure compares AMOC curves for otherwise identical surveillance systems that included either any ARI outpatient visits (red circles, using the EARS W2c statistic [2] and blue triangles, using the CUSUM statistic) or pneumonia (blue triangles, using the CUSUM statistic modified for sparse data [3]). Note that Detection Delay (y-axis) is lower at any given FAR when surveillance aims at patients with pneumonia. Sensitivity analysis suggests that this advantage remains true when pneumonia complicates influenza ≥ 5% of the time.


Our results suggest that EMR-based influenza surveillance that targets patients with pneumonia can outperform systems that monitor all ARI patients.

[1]. DeLisle S, South B, Anthony JA, Kalp E, Gundlapalli A, et al. Combining Free Text and Structured Electronic Medical Record Entries to Detect Acute Respiratory InfectionsPLoS ONE 2010;5(10):e13377.
[2]. Hutwagner L, Thompson W, Seeman GM, Treadwell T. The bioterrorism preparedness and response early aberration reporting system (EARS)Journal of Urban Health: Bulletin of the New York Academy of Medicine 2003;80(2) (Suppl. 1):i89–i96.
[3]. He S, Huang W, Woodall WH. CUSUM charts for monitoring a zero-inflated Poisson processQual. Reliab. Engng. Int 2012;28:181–192.

[Figure ID: f1-ojphi-05-38]

Article Categories:
  • ISDS 2012 Conference Abstracts

Keywords: influenza, surveillance, modeling, simulations.