Agreement Behavior of Isolated Annotators for Maintenance Work-Order Data Mining

Abstract

Maintenance work orders (MWOs) are an integral part of the
maintenance workflow. These documents allow technicians
to capture vital aspects of a maintenance job: observed symptoms,
potential causes, solutions implemented, etc. These
MWOs have often been disregarded during analysis because
of the unstructured nature of the text they contain. However,
many research efforts have recently emerged that clean
these MWOs for analysis. One such effort uses a tagging
method with an open source toolkit, named Nestor, which relies
on experts classifying and annotating the words used in
the MWOs. For example, an expert might classify the words
“replace,” “replaced,” and “repalce” as “Solutions” and give
the alias “replace” to all of them. This method greatly reduces
the volume of words used in the MWOs and links words,
including misspellings, that have the same or similar meanings.
However, one issue with the current iteration of this
tool, along with practical usage of data-annotation tools on
the shop-floor more generally, is the usage of only one expert
annotator at a time. How do we know that the classifications
of a single annotator are correct, or if it is, for example, feasible
to divide the tagging task among multiple experts? This
paper examines the agreement behavior of multiple isolated
experts classifying and annotating MWO data, and provides
implications for implementing this tagging technique for use
in authentic contexts. The results described here will help improve
MWO classification leading to more accurate analysis
of MWOS for decision-making support.