Extending an Uncertainty Taxonomy for Suspected Pneumonia Case Review

Abstract

We present our efforts towards extending a taxonomy of uncertainty for suspected pneumonia case review. We manually annotated a random sample of 200 cases of suspected pneumonia at the case level, classifying cases as suspected, unlikely or cannot be determined. Within each document, evidence snippets were annotated that support, refute or are uncertain for suspected pneumonia. Finally we conducted a thematic analysis to classify mentions marked as uncertain applying a taxonomy of uncertainty. We observed substantial variability in human agreement at the evidence snippet level but high agreement for classification of evidence snippets to a taxonomy of uncertainty.