Best Practices Framework for Improving Maintenance Data Quality to Enable Asset Performance Analytics

Field maintenance data is often captured manually and is prone to have incomplete and inaccurate information in the structured fields.  However, unstructured fields captured through work order planning, scheduling, and execution contains a wealth of historical information about asset performance, failure patterns, and maintenance strategies.  The prevalent data quality issues in maintenance data need to be understood and processed in order to extract actionable intelligence.  This paper describes a best practices framework for measuring and improving data quality, developed through years of research and working with 120+ process and manufacturing organizations.  The framework enables evaluating and executing analytics by identifying strengths in the data.  It determines where and how asset performance measures such as benchmarking metrics, reliability measures, and bad actor identification can be evaluated with confidence.  Missing or inconsistent information can be extracted from the unstructured fields using natural language processing (NLP) techniques to bridge gaps in the analysis.  While the NLP algorithms make historical data usable for some analytics, the best practices identify improvements in the work process of capturing data, thereby improving future quality.  A feedback on data quality indicators completes the loop to sustain improvements.


INTRODUCTION
Many industrial organizations would like to take advantage of emerging technologies and data-analytics to meet their business objectives, but are challenged by the quality of their maintenance data.Field maintenance data generated and stored in centralized computer systems such as an Enterprise Asset Management (EAM) or a Centralized Maintenance Management System (CMMS) contains a wealth of information around maintenance history, failure patterns and the performance of industrial assets under different operating conditions.There is potential to integrate information from maintenance data into work processes for managing asset performance, but the raw data by itself is challenged by data quality problems at nearly all industrial organizations.Poor data quality is a common issue for every process plant in the world, and can be a significant limiting factor in performing accurate analytics such as for determining asset performance, for predicting failures, and for determining optimal maintenance strategies.
The challenges in the quality of field maintenance data is not only on the completeness or the general quality of the data, but also in the configuration and the usage of the CMMS/EAM.A major challenge lies in the culture of the organization and the motivation in capturing accurate data.The intent is not in achieving perfection in every aspect of a maintenance work order, but to reach a "sufficiently-good" state.In order to reach a "sufficiently-good" state, the objectives for how the data should be used the context of business goals need to be clearly defined.Data quality is only meaningful in the context of desired outcomes.This paper describes a best-practices framework for measuring and improving data quality of maintenance data from the CMMS/EAM for applications related to analytics measuring asset performance and reliability.The data quality improvement framework is a component of a larger work process for executing asset performance analytics in a scalable way across general datasets.For industrial companies who wish to adopt and integrate data-driven analytics into asset performance management (APM) work processes, the authors recommend two parallel initiatives: while implementing APM initiatives, simultaneously implement the data quality improvement work process.
The first step in the data quality improvement process is to measure data quality.Data quality measurements are used to determine which data is sufficiently-good for analysis, and which data needs improvement.After this assessment, the data that is sufficiently good can be used immediately to evaluate asset performance analytics.The data quality assessment governs which analytics are possible to evaluate with sufficient confidence levels.For data fields required for desired analytics where the data is too poor to use, it is necessary to improve the quality.At this point, baseline quality measures have been established as well as a clear identification of where to improve.Key to improving data quality is to fix the work process to fix future data.By implementing best practices in the usage of the CMMS/EAM and creating a feedback loop to measure and track the quality of new data, future data can be refined and made suitable for the purpose of analytics.Culture change is often slow, but the complimentary nature of making assets more reliable and reducing the effort spent on avoidable maintenance can be a boost to making a change in work processes.
Additionally, there is opportunity to improve data quality by identifying other data sources containing complementary desired information and applying analytics to integrate that information into the asset performance management work processes.In particular, the data needed for many common reliability metrics and analytical methods such as mean time between failure (MTBF) and Weibull analysis is often missing from the structured fields but can be systematically extracted or inferred from the unstructured description fields using tools from natural language processing (NLP) and machine learning.In these cases, NLP and machine learning can be implemented to improve historical data so it can also be used immediately for analytics.This paper describes the framework and the various framework components including assessment and tracking of data quality, well-defined asset performance metrics, analytic tools from NLP and the implementation of best practices in an industrial organization.The rest of this paper is organized as follows.Section 2 reviews and characterizes data from the CMMS/EAM, different data quality challenges and maintenance management processes, as well as reviews efforts to date for using NLP approaches on industrial maintenance data.Section 3 provides the theoretical work process, and Section 4 presents a case study illustrating the end-to-end framework.The paper ends with concluding discussions and suggests future research directions.

BACKGROUND
This section covers background and literature review on characterization of maintenance data from the CMMS/EAM, uses for CMMS/EAM data beyond its intended purpose, and a summary of studies applying NLP on unstructured fields in maintenance data for different industrial applications.

Maintenance data
Understanding the maintenance management work process, which includes work task identification, planning, scheduling, and reporting is important for understanding and characterizing field maintenance data.Maintenance data creation and storage is typically generated in a maintenance management system such as an EAM or CMMS (Gulati & Smith, 2013).Capabilities of CMMS/EAM include anything around a work order for maintenance such as assigning personnel, materials, recording costs, and tracking information history (Tretten, 2014).A typical maintenance process described by Gulati and Smith (2013) is shown below in Figure 1.There are several individuals involved in the maintenance management process in addition to the technician actually performing the maintenance work including coordinators, planners, schedulers, and craft supervisors.First the work request is routed to the asset/resource coordinator who prioritizes the work and sends it to the planner or scheduler.Maintenance planners plan the job and create a work plan while maintenance schedulers work with the craft supervisor to develop the plans of when the different work will be executed.The supervisors then assign who will do each job and ensure that high quality work is maintained and the details of work are properly recorded and documented in the CMMS/EAM.

Figure 1
Typical maintenance management work process of maintenance, reproduced from Gulati and Smith (2013).Different players all have roles in the process, which is managed through a centralized CMMS/EAM.Databases from CMMS/EAM systems include records of maintenance and maintenance costs across asset fleets, for failure management, equipment management and budget management (Suzuki, 1994).CMMS/EAM systems are typically designed to support business performance work processes such as work order management, accounting, and procurement.Benefits of a CMMS/EAM in an organization to support the maintenance management process include enabling data sharing and improved communication through enterprise wide access to one digitalized maintenance system, efficiency through reduction of personnel and steps, improved accountability and response time, tracking processes and inventory, and to improve maintenance work and budget management through managing resources.
Over time, the use of field maintenance data generated by the CMMS/EAM system has expanded beyond maintenance management.Historical maintenance records can provide valuable insight into past maintenance on pieces of equipment.One widely-used standard in the oil and gas industry for the collection and exchange of reliability and maintenance data is ISO 14224 (ISO 2004).ISO 14224 categorizes reliability and maintenance data into three categories: equipment, failure, and maintenance.Equipment data contains the description of the equipment, which includes attributes, data about operating context, and hierarchical classification.Failure data is data used for identifying and characterizing failure events, and maintenance data is data for characterizing maintenance events.The relationship between maintenance data and failure data is that corrective maintenance events are recorded to describe the corrective work event following a failure event.
The use of maintenance data for reliability purposes is not new, but recently technological advances in data storage, computational capabilities and data-driven analytics have generated both the potential and the surge of interest to develop and implement work processes with stronger emphasis on data utilization.Other data sources about an asset in addition to data from the CMMS/EAM include timeseries data from condition monitoring, information from the original equipment manufacturer (OEM) or vendor such as recommended maintenance, piping and instrumentation diagrams (P&ID), inspections, as well as deep knowledge possessed by the operators and maintainers themselves which could possibly be captured in the form of failure modes and effects analysis (FMEA) templates or maintenance strategies.There are potentials for utilizing information found in other data sources to complement information from the historical maintenance data in data-driven analytics for nonmaintenance purposes such as reliability.
Several studies have explored various dimensions for measuring data quality (Aljumaili, 2016) (Koronios et al., 2005) (Araújo, 2016) (Chambers, 2016) (Cohen, 2017) (Lemma, 2012) (S.J. Lin 2006) (Loshin, 2009) (Loshin, 2011) (Sun 2011) (Distefano & Thomas, 2011).Different industrial companies use the CMMS/EAM in different ways, which leads to differing fields and ways to capture the data, leading to hurdles surrounding standardization.Standardization is a requirement needed to utilize content around industrial data analytics, but there are many levels of standardization needed to get there.The work processes that generate maintenance data should be standardized with respect to how the CMMS/EAM is used across multiple sites or companies.Standardized definitions for creating, planning, scheduling and executing maintenance work orders are needed for the entire maintenance management work process.Additionally, standard codes, and standardized location hierarchies are also needed (Naik & Saetia, 2018).Hodkiewicz, Kelly, Sikorska, and Gouws (2006) notably developed an eight-step framework for assessing and improving data quality for reliability analytics.This framework is aimed at implementing best practices for improving future data.Their eight-step framework starts with identifying the business need, identifying the metrics which support the business need, identifying the data required for each metric, and then analyzing the data quality of each data element.Once the data quality has been systematically meared, the next step is to analyze how to improve the quality of each data element.This analysis determines the changes needed to implement in how the data is captured.Once changes have been implemented, assess the improvements and execute a periodic review process in the context of the business need.This framework provides an excellent breakdown of the steps needed to bridge the connection between the business need and the nature of the data itself, and further incorporates best practices where appropriate as well as continuous tracking.
In terms of data quality, ISO-14224 recommends using both pre-set codes and unstructured fields in maintenance data, because often information can be missed from a code.In many cases, the true nature of the failure cause can be inferred from unstructured fields such as the free text field or service notes.Recently, there has been a surge of use of methods from natural language processing (NLP) towards getting information from unstructured maintenance fields.Different approaches both through analytics to clean existing data or processes to improve data collection have been discussed and explored and will be reviewed in the following section.

NLP for field maintenance data
In this section, different studies that have used NLP on the unstructured free text fields in field maintenance data are reviewed in the context of the different applications and outcomes rather than from the perspective of the different approaches/techniques utilized.One commonality between all the applications covered below is that they are all for purposes beyond maintenance management.

Asset performance metrics and benchmarking
Maintenance performance metrics (MPM) are metrics that are used to measure the efficiency and effectiveness of maintenance strategies and frameworks (Parida & Kumar, 2009).There are many categories or pillars of metrics used in industry that have been extensively reviewed (Kumar 2013) and standardized definitions for best-practice have been established by the Society of Maintenance and Reliability Professionals (SMRP) (SMRP Best Practices 2017) and the European Federation of National Maintenance Societies (European Federation of National Maintenance 2007).Benchmarking is the processing of identifying, sharing, and using knowledge and best practices and can be used in conjunction with metrics to identify opportunities and measure benefits of an asset performance improvement initiative.
Accurate evaluation of metrics is fundamentally challenged by data quality problems.Poor data quality can erroneously alter many common metrics to look good.For instance, not recording failures properly can improve measures of asset reliability.Inconsistent data recording practices can result in inconsistent values when trying to implement comparative analytics.Unstructured fields can be used to both populate missing fields and to provide a way to improve consistency in structured fields which are populated.Naik (2015) described a work process to identify equipment having chronic failures through using benchmarking to identify opportunity relative to peer groupings.Once a bad actor was detected, the information in the work order description was necessary for understanding the nature of the identified chronic failures.Brundage, Sexton, Morris, Moccozet, and Hoffman (2018) proposed a set of maintenance key performance indicators (KPI's) based on the characterization of the data from the CMMS/EAM and the possibilities in the data characterizations after incorporating results from NLP on the unstructured fields.Lukens and Naik (2019) proposed a methodology for practitioners to consistently utilize the SMRP Best Practices metrics with respect to data quality considerations, leveraging NLP technologies where appropriate.Traditional maintenance KPI's have been developed from a functional perspective (and rightly so), but in practice have been challenged as field data gets wrangled to fit metric definitions.The approach of defining or identifying how to evaluate KPI's robustly and consistently from the perspective of the data is a novel concept with increasing importance to compliment the increasing emphasis on data-driven technologies in maintenance processes.

Reliability analytics
Reliability analytics encompasses different analytical methods concerned with the characterization of failure events used for the estimation, prevention, and management of failures in reliability applications.This class of analytical methods can range from simple analyses based on failure characterizations such as Pareto charts to methods in survival analysis such as Weibull analysis or repairable systems analysis.The first step towards assessing reliability using a dataset is understanding which data describes failure events.The Society for Maintenance and Reliability Professionals (SMRP) defines a failure as a situation when "an asset is unable to perform its required function."(SMRP Best Practices 2017).Despite this clear definition, in practice many engineers and operators hesitate to mark an asset as failing when an incident occurs.
There have been several studies using NLP on the unstructured text to characterize event types.Edwards, Zatorsky, and Nayak (2008) used an unsupervised clustering approach (similar to topic modeling) to determine if maintenance work orders were scheduled or unscheduled.While they were unsatisfied with their results towards achieving that goal, they were successful in identifying 14 distinct of maintenance jobs in their dataset which could be useful for analyses requiring failure event classifications.Bastos, Lopes, & Pines (2012) developed a knowledge mining framework which incorporated data mining on unstructured free text for maintenance teams to use for forecasting failures, and tested it on determining if an event was preventative or corrective.Arif-Uz-Zaman (2016,2017) used Naïve Bayes classification to determine if a work order (for Australian power plant data) was a failure or preventative maintenance.Naik and Saetia (2018) used a classifier to determine the event type, because the event type code for their use case was used too generically to distinguish between corrective and preventative work.
A Naïve-Bayes classifier was developed on labeled data from the CMMS/EAM from an aggregate database from many industrial facilities from around the world and used by the authors to determine if a corrective work event from the CMMS/EAM was a failure or not (Lukens & Markham, 2018a) (Lukens, Naik, Markham & Laplante, 2019).The labeled training database was curated from working with a team of domain experts who reviewed and labeled if a repair description described a failure or not.Naïve-Bayes was selected as the classifier because it was the lazy learner that had the best performance based on the performance criteria, and the project required the flexibility for new users to add data and retrain the model if needed while preserving the anonymity of the aggregate database.
Once the failure is identified, the next level of reliability analysis is basic failure event classification.Stenström (2015) looked at term frequency of a corpus of maintenance work orders on rail infrastructure to gain insights on common or repeated failure patterns.The process of looking at terms and tagging similar words based was developed further by Sexton, Brundage, Hoffman and Morris (2017), who formally introduced a tagging work process for creating content on maintenance work orders which could be used very powerfully and simply.They further suggested some simple applications such as plotting "hot spots" of common failure classifications over time.He (2016) integrated machine learning classification using unstructured fields into a data unification work process towards using data for data analytics.
Conducting reliability analytics such as Weibull analysis or failure forecasting on general maintenance data has traditionally been challenged by data quality issues, but enabled with the application of NLP on the description fields.Hodkiewicz and Ho (2016) used a rule-based approach for work orders on heavy mobile equipment in mining, and makes several suggestions from insights gained in the study on how maintenance data should be entered or created in the first place into the CMMS/EAM.Sexton, Hodkiewicz, Brundage, and Smoker (2018) compared failure characterizations between rules-based and word tagging through survival analysis and fitting Weibull distributions.Lukens and Markham (Lukens & Markham, 2018b) did Crow-AMSAA modeling on characterized failures in order to look at trends in the recurrence of different failure modes over time.

Applications for reliability information
One application or use for the results of reliability distribution fitting and maintenance metrics is system reliability modeling.Parameters fit from reliability models can be used as inputs to reliability simulations of different systems over time.The results of reliability-based simulations can be used to estimate the contribution of the total cost of ownership from maintenance and reliability over a product's lifecycle which can be used to make procurement or repair versus replace decisions based on data.Hodkiewicz, Batsioudis, Radomiljac, and Ho (2017) performed a system reliability analysis for mining shovels to provide quantitative information for a business case towards automating shovels.Characterizing shovel failures from maintenance data was a necessary step in this analysis because it provided the granularity to partition which maintenance events could be avoided in an automated system.Lukens, Naik, Markham, and Laplante (2019) focused on understanding annual costs and reliability from different failure modes from observed field data in order to illustrate the challenges that need to be considered to extract actionable insights to use maintenance data across a supply chain.Mahlamaki, Niemi, and Jokinen (2016) did simulation modeling for estimating maintenance life cycle costs of a product.
Another application which uses reliability and failure mode information is the development of maintenance strategies.
Traditional approaches include reliability centered maintenance (RCM) and failure modes and effects analysis (FMEA).There is a massive opportunity for incorporating historical data into the maintenance strategy management creation, deployment, and tracking process.Sikorska, Hammond and Kelly (2008) develop a data-cleaning approach for reliability analytics from CMMS data using case-based reasoning (CBR).They created a search tool called "failure finder" to find likely failure modes associated with a record, to link an RCM database with maintenance data from the CMMS.Yang, Letourneau, Zaulski, and Scarlett (2010) used maintenance data from Auxiliary Power Unit Engine (APU) to quantify instances of part replacements, using the unstructured text to match replacements with the manufacturer-supplied FMEA to compare observed with the manufacturer recommendations.As a result, a workflow to use real world data to assist in decision-based system to assist in the maintenance of an asset was proposed.
Lukens and Markham (2018b) described a high-level workflow for CMMS/EAM data for identifying, prioritizing, and creating asset strategies systematically, providing examples of how data-driven analytics can augment work processes for initiating RCM programs.Lukens and Markham (2018a) showed how natural language processing can be incorporated into traditional approaches for calculating age-failure patterns across different asset populations in order to assess the relationship between reliability of a group of assets and the operating age, and to suggest strategies based on this information.

Other industrial applications
The use of NLP on industry data can go beyond characterizing maintenance work orders for maintenance and reliability applications.Brundage, Kulvantunyou, Ademujimi and Rakshith (2017) developed a knowledge framework for root cause analysis in manufacturing where NLP was used to create a knowledge database.Saetia, Lukens, Hu, and Pijcke (2019) used the equipment short description in the equipment registry to characterize a standard taxonomy for equipment, enabling matching "offthe-shelf" analytics to relevant equipment.Nair (2018) developed a workflow for inspections of fixed equipment in order to rapidly make data-driven decisions during a turnaround.In the workflow, inspection data was mined as it was entered for potential failures which were compared to other data sources such as thickness measurements to determine if action was needed during the turnaround.

THEORETICAL FRAMEWORK
In this section a best practices framework is presented for improving maintenance data quality as part of enabling asset performance analytics.High-performing organizations who wish to adopt data-driven analytics in work processes for improving asset performance should do so in the form of two parallel initiatives: improving data quality and asset performance analytics.Both initiatives are integrated in the framework.
In order to measure the quality of the data, it is important to have clearly defined business goals.In this general framework, the business goals are generically defined as those of an APM initiative such as improving asset performance and reliability through reducing unplanned downtime, maintenance costs, etc.However, the exact objectives depend on the organization.
The first step in this exercise is to quantify the data in terms of areas of "sufficiently-good data" and data that requires improvement.The areas where the data quality is identified as "sufficiently-good" can be used right away to kick off the asset performance analytics initiative such as through generating certain metrics and identifying poor performing assets (bad actors).Parallelly, for the identified poor data quality, start putting work process improvements in place that ensure that the bad areas of data turn into good areas, which can then feed into the asset performance analytics.Lastly, it is important to monitor and track the effort.The process is summarized below in Figure 2, and each step is described in the more detail in the following sections.utilizing field maintenance data for asset performance analytics.Data that is "sufficiently-good" for asset performance analytics can be used immediately, while work processes to improve poor data can be put into place.

Data quality assessment
Identify the areas where the data is "sufficiently good", and where the data needs improvement.The concept of "sufficiently good" is congruent with the definition of quality data as "data that is fit for purpose" (Hodkiewicz et al. 2006).
To measure the quality of the data it is important to first define what you are trying to measure with the data, aligning with business goals.This process is explicitly spelled out in the first four steps in framework developed by Hodkiewicz et al. (2006).
To apply this framework to generalizable asset performance metrics and analytics, first there must be a mapping between the maintenance management process and how the data is captured at an industrial organization to a standard view of the maintenance processes.In this way, data fields can be mapped to standard definitions of event types, different failure codes, standard equipment taxonomies, etc.Standard ways of assessing different maintenance performance measures create a consistent and repeatable mapping between CMMS/EAM data and the data quality assessments.Assuming a standardized scorecard of desired asset performance metrics and analysis, the quality of all of the different data fields needed for each measure can be measured with a summary report.The scorecard summary will show which metrics can be estimated with "sufficientlygood" confidence, and which ones cannot be evaluated because of poor data.From here it is the possible to drill down, identify, and measure the precise factors which drive these measurements.
For example, if an organization is good at capturing failure dates, out-of-service and in-service dates but bad at identifying the total maintenance cost, then the recommendation is to rely on mean time to repair (MTTR) or asset downtime to identify bad actors rather than maintenance cost.More specific examples are described in detail in the case study.

Improve historical data
For data fields required for desired analytics where the data is too poor to use, it is necessary to improve the quality.Improving historical data refers to improving the quality of existing data through incorporating data analytics to triangulate missing or inconsistent information from the data using other data.For asset performance measures specifically, the information in the unstructured field often provides detailed information that is commonly missing such as event type, failure mode, or component which characterizes the data.For certain fields that can be filled in through NLP, NLP analytics are generated to fill them in.
One major strength of using analytics to populate structured fields is consistency.Two similar inputs will always have the same classification by a computer model, while two similar inputs may have different codes if two different humans are filling in the work order.Another strength is speed and scalability.What may take a human days or weeks can take seconds or minutes running a computer model.Some data fields are better candidates than others to be filled in through NLP such as breakdown indicator and failure mode.However, other data fields are more challenging such as dates, costs, detection methods, etc. Evaluating the strengths (and data quality) of the unstructured data is part of the feedback loop.

Implement best practices to improve future data
Implementing best practices to improve future data refers to fixing the work process to improve the quality of future data as it is generated.Where to fix the work process depends on where the desired data quality improvement areas which have been identified in previous steps are located.It extremely important to have feedback loops in place that monitor data as it comes in and flags exceptions.Humans generally tend to want to do a good job if they know that someone actually cares about the data they enter into the system.A connection between the people entering the data and the people analyzing the data needs to be made.By having accuracy in the former, the latter can pinpoint the underlying causes and improve the performance of the assets, which in turn will make it easier for the former as assets break down lesser and lesser.
A recommended work process for improved best practices is through identification and appropriate modifications in the maintenance management work processes.First, there needs to be a work process in place for tracking data quality, which depends on what data quality measures you have identified to track.Simple dashboards containing summary information such as work orders with missing data can provide visibility into specific work orders that have been closed recently and are missing data.
Gulati and Smith (2013) describe a typical work maintenance process as shown in Figure 1.However, what is missing from the traditional maintenance process is monitoring and tracking data quality.As a best practice, one suggestion is that the work process should be adapted as per Figure 3 below, introducing the role of a data quality reviewer.The data quality reviewer analyzes data quality reports and the dashboard each month and follows up with the right people if certain information is missed.

Figure 3 Proposed maintenance work process with data quality reviews
The framework presented above can be generalized for applications beyond asset performance analytics.Certain specifics such as integrating NLP to extract structured information can be generalized to any triangulation information using other data sources.For any application, an essential component is standardization of data into some form for repeated use which is not trivial.For APM applications, the authors have developed and deployed this work process repeatedly specifically using standards developed for maintenance and reliability.Mapping data into such an analytic template enables repeatable and scalable use of various analytics across different industrial companies.A case study on a maintenance data set is presented in the next section to illustrate the end-to-end specifics of the theoretical framework in the next section.

CASE STUDY
An end-to-end example of the proposed data quality measuring, improving, and tracking process is presented step by step in the following case study.The case study applies to a generic asset-intensive company with multiple sites where maintenance management is conducted through a CMMS/EAM.The company wants to begin an APM initiative for managing and improving the performance of their assets, using their historical data for asset performance analytics such as benchmarking KPI's, identifying bad actors to focus on for optimal maintenance strategies, and for determining the root cause of chronic failures.
To protect proprietary information, all variables have been anonymized and age has been scaled.This case study is for illustrative purposes only and values do not reflect any one specific company.We restrict the scope of the case study to a subset of data for a single equipment type (such as a centrifugal pump) at comparable sites by industry, size, and region of the world.The dataset identified for the study consists of N = 1,800 assets observed over a period of 4 units of time (such as a year) for 3 different sites.There were about 8,000 repairs observed total over that period of time.
Existing standard metric definitions for reliability benchmarking established by the Society of Maintenance and Reliability Professionals (SMRP) are used as a starting point (SMRP Best Practices 2017).The company determined that a scoreboard reflecting their objectives could be realized with evaluating Mean time Between Failure (MTBF), Average Corrective Work Cost ($), and Reactive Work Count (%).
Standard definitions with descriptions of the type of data needed for each of these metrics are reported in Table 1.After understanding the standard definitions, the need was to define how to apply data consistently and in a scalable way to these definitions, a process that is not straightforward.The process for applying data consistently to standard definitions is challenged by the fact that many existing standards are theoretical and not designed with consistent rules for applying data in mind.

Measuring data quality
Once the definitions with respect to the data were clear, a summary report is produced which reports the data quality in Table 2.The summary values of "Low", "Medium", and "High" are aimed at describing the level of "sufficientlygood" or needing improvement of the data for evaluating each metric.The different data quality factors needed for each metric are defined and measured, and combined to produce the "Low", "Medium" and "High" values.The high score for Reactive Work Count (%) was due to the fact that the priority field was well populated.In general, priorities tend to be well-populated in maintenance data because they are required fields in many popular CMMS/EAM.The goodness of how consistently a priority score is used from site to site is an open question to determine (for instance, benchmarking reactive work between two sites could be challenged by inconsistent definitions of what is an emergency).

Good quality data
The best practices workflow recommends identifying where the data is sufficiently-good and using that data immediately.
In the case study, while it was identified that evaluation of the MTBF metric and reliability analytics are challenged by the fact that the company does not record failures, it was identified that that they were decent at recording which maintenance work events were repairs, or corrective work, and which were reactive work (emergency repairs).
In this example, the MTBF evaluated for the pumps using the raw maintenance data came out to 48,000 months, a nonsensical number.However, using the decent data, other metrics such as mean time between repairs (MTBR) and the mean time between emergency repairs could be calculated.The values of these three metrics are reported in Table 3.In this example, while MTBF and MTBR are well-established metrics, the metric "mean time between emergency repairs (MTBER)" was made up based on the data quality and functional purpose.Mean time between emergency repairs (MTBER) 37

Improving historical data
After investigating corrective work orders, several work orders that should have been marked failures according to the SMRP definition of failure but were not correctly marked in the maintenance management system prevented an accurate calculation of the MTBF metric (Table 3).However, in addition to identifying fields that were decent to use (priority and event type), it was also identified that the description fields contained a rich amount of information which outlined the problem and key relevant information.In  In this situation, it is possible to use analytics with the description field to improve the historical data by identifying which event is a failure event using the description.In the case study, we use the same machine learning classifier trained on labeled data in (Lukens & Markham, 2018a) (Lukens et al., 2019).The machine learning classification model uses the text field as input data and returns a prediction (and prediction score) on whether that event was a failure or not.Users go from having missing of "unknown" fields to performing analyses on detailed data for insights that help organization improve asset performance.
The company with an MTBF of 78,000 months applied this classifier to it data and the MTBF was recalculated to be 15 months.Further, before there was not enough information before to benchmark MTBF between the 3 sites, but afterwards the capability to benchmark between sites in such a way that the model was consistent in what it characterized a failure or not.A comparison is shown in Figure 4.
Figure 4 Benchmarking reliability through mean time between failures (MTBF) between sites for an asset type.
Once an infrastructure for text mining analytics has been built around the maintenance data, from the work descriptions it is typically possible to characterize additional key information for reliability analytics such as failure mode and maintainable item.For instance, it becomes possible to benchmark information at the failure mode level, and to evaluate the maintenance strategy based on risks and risk mitigating actions.In the case study, it was determined that the most common failure groupings were seal failures, bearing failures, and valve failures.Comparisons of MTBF between the two sites by failure groupings are shown in Figure 5. Observe that in both cases described (identifying metrics that can be evaluated given the current state of the data quality and incorporating NLP to fill in information for evaluating other metrics), the data quality workflow enables confidence in the metric value.Strengths of the analytics approach is that the model is always consistent while data may be captured differently between different sites.However, strengths in using "proxy metrics" include transparency -analyses into "why" certain numbers are straightforward without needing to understand a model.

Improving future data
The next piece is to improve and track improvements to future data through introducing best practices at how the data is generated.While the company can continue to rely on metrics using "proxy metrics" or by implementing NLP on their maintenance work orders, there is need to also start putting work process improvements in place to ensure that the bad areas of data turn into good areas.
In the case study, the company also wanted to evaluate maintenance costs, which are not as straightforward to fill in from text mining.In their case, they identified that they were only using costs about 20-30% of the time across their plant.
The reliability engineers identified several issues in their cost accounting work process, and they began to start fixing their work process in accounting for the total maintenance cost: including labor hours, materials, and contractor costs.With constant monitoring and tracking of work orders with missing costs, they were able to reverse the trend of missing costs within three years since they started monitoring and tracking their data quality.The results of the data quality tracking are shown in Figure 6.An end-to-end example starting with raw maintenance data from the CMMS/EAM for a generic industrial company was presented.The business goals were defined with what the company wanted to achieve by utilizing their data, and what they wanted to measure.Data quality was then systematically measured towards executing these analytics, and based on where the data was sufficiently good, metrics were identified based on the desired goal.Text mining analytics was incorporated to the process in order to utilize the free text description fields in the historical data in a consistent scalable way.Lastly, because exactly what areas data quality was needed was precisely defined, the company was able to identify challenges with their existing data collection practices and introduce best practices.Through a process of measuring and tracking data quality, the company is able to have confidence in the outputs of their analytics towards meeting the business objectives.

CONCLUSION
Data quality issues are prevalent and wide-spread across industrial organizations, but without knowing specifics, the use of data for analytics and decision making should not be dismissed or postponed.A framework for measuring data quality executed simultaneously with efforts to utilize data for asset performance analytics was presented, and a case study developing specific details end-to-end was presented.
There is much opportunity to build out the process for industrial applications beyond maintenance and reliability, as well as integrating other common data sources.Key steps for integrating other data sources includes understanding how a new data source can be used to meet business goals for industrial companies, and developing standards for using the data source in an analytics template.For prognostics and health management (PHM) work processes, understanding failure mode and health monitoring patterns for different assets is key.
One key root cause of the data quality challenges prevalent in CMMS/EAM data is the maintenance management work process itself.The best practices workflow outlined here provides a process for identifying where in the existing maintenance management process to focus on improved data quality.Stepping back even further, there is opportunity to re-evaluate the entire maintenance management process with respect to generating and collecting reliable, consistent data for the ready use of digital solutions.Important to note is an emerging research effort focused on developing a framework for evaluating existing maintenance management processes which are dominated by human factors, starting with identifying sources of human unreliability (Brundage, Sexton, Hodkiewicz, Morris, Arinez, Ameri, Ni & Xiao, 2019).Once sources of human error in the maintenance management process are systematically identified and quantified, the maintenance management process itself can be re-evaluated.The direction of this research effort and future adoption specifics by industrial companies is something to watch and will be very important as more emphasis is placed on integrating data-driven and machine technologies into the industrial space.

ACKNOWLEDGEMENT
The authors acknowledge Erik Pijcke, Matt Markham, Vincent Yates, Shaddy Abado and Roy Whitt for contributing their expertise to many constructive conversations in developing the framework we use today.

Figure 2
Figure2Data quality best practices workflow presented for utilizing field maintenance data for asset performance analytics.Data that is "sufficiently-good" for asset performance analytics can be used immediately, while work processes to improve poor data can be put into place.

Figure 5
Figure 5 Benchmarking reliability by failure grouping

Figure 6
Figure 6 Data quality tracking for missing costs.Once the company began implementing best practices, monitoring and tracking data quality improvements showed an improvement from about 20% of work orders with costs filled in to over 90%.

Table 1
Metric definitions and description of data needed for maintenance and reliability metrics desired in the case study.

Table 2
Scoreboard for case-study company summarizing data quality with respect to desired metrics defined in Table1.In this example, MTBF was low at all sites because the company did not use the breakdown indicator, a field in the CMMS/EAM used to record whether the asset failed or not.In fact, of the 8,000 repairs in the dataset, only 2 of them were marked as failures.The reason for low score for average corrective work cost was dominated by many missing costs at one site.

Table 3
Summary metrics for one asset type across a multisite industrial company in the case study

Table 4 ,
work orders with IDs 1-4 should have been marked with the breakdown indicator set to TRUE instead of FALSE in the CMMS/EAM to indicate asset failure and be counted in the MTBF metric.

Table 4
Sample repair work orders for the case study for pumps WO