Probability of Detection (POD)-based Metric for Evaluation of Classifiers Used in Driving Behavior Prediction
Classiﬁers are functional tools/algorithms that implement classiﬁcations and are widely used in science and technology for state of health estimation, diagnosis systems, and situation/intention recognition of human operators. Certiﬁcation of these classiﬁers plays a crucial role in their selection for a speciﬁc task. Current certiﬁcation approaches utilize the Receiver Operator Curve (ROC) as a standard tool that provides graphically the performance of classiﬁers. Beside the ratio of Detection Rate and False Alarm Rate (combined as ROC), other properties related to process parameters are not considered. In this paper, a new evaluation method based on the Probability of Detection (POD) reliability measure is developed discussing the effect of further process parameters on the classiﬁcation results. Probability of Detection (POD) serves as a performance measure for quantifying the reliability of conventional Nondestructive Testing (NDT) procedures and Structural Health Monitoring (SHM) systems. The approach considers statistical variability of sensor-based measurements. In this publication for the ﬁrst time the signal-response and the binary (hit/miss) approaches are implemented in combination with a process parameter. As example in this publication, the prediction of driving behavior classiﬁcation is used as process parameter. The signal response approach is applied to compare the driving behavior prediction capabilities of Fuzzy Logic-Hidden Markov Model (FL-HMM), Artiﬁcial Neural Network (ANN), and Support Vector Machine (SVM) with respect to the reliability of the prediction for driver behavior related to prediction time. The hit/miss approach is also applied on FL-HMM as example for predicting an upcoming driving maneuver. To account for data uncertainty and variability, conﬁdence bounds are established. A typical and useful criteria for detection at a 90 % probability of detection level with 95 % conﬁdence level is successfully implemented as a new reliability measure and certiﬁcation standard for classiﬁers. In this article a new approach is established permitting a new evaluation approach to classiﬁers. The new approach introduces a POD-based measure for comparison of binary classiﬁers.
How to Cite
Probability of Detection, Classifiers, Situation/Intension recognition, Driving behavior prediction, Artificial Neural Network, Support Vector Machine, Fuzzy-logic Hidden Markov Model, Reliability certification
This work is licensed under a Creative Commons Attribution 3.0 Unported License.
The Prognostic and Health Management Society advocates open-access to scientific data and uses a Creative Commons license for publishing and distributing any papers. A Creative Commons license does not relinquish the author’s copyright; rather it allows them to share some of their rights with any member of the public under certain conditions whilst enjoying full legal protection. By submitting an article to the International Conference of the Prognostics and Health Management Society, the authors agree to be bound by the associated terms and conditions including the following:
As the author, you retain the copyright to your Work. By submitting your Work, you are granting anybody the right to copy, distribute and transmit your Work and to adapt your Work with proper attribution under the terms of the Creative Commons Attribution 3.0 United States license. You assign rights to the Prognostics and Health Management Society to publish and disseminate your Work through electronic and print media if it is accepted for publication. A license note citing the Creative Commons Attribution 3.0 United States License as shown below needs to be placed in the footnote on the first page of the article.
First Author et al. This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 United States License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.