TireEye: Optical On-board Tire Wear Detection

Automotive tire tread depth has a major influence on a car's safety and thus has to be closely monitored. However, there is currently no on-board solution that can measure tire wear with an accuracy of less than 1 mm. TireEye is an optical device mounted inside the wheel well and facing towards the road. It records the cross-section of the longitudinal tread groove and extracts its outline by means of adaptive Canny edge detection. The tread wear indicators serve es means to calibrate the scale since they provide the minimum tread depth.


INTRODUCTION
Tire tread wear is inevitable when driving a car. The tread depth of new tires is about 8 mm, depending on the manufacturer and tire model. The most minor permitted depth is 1.6 mm in most countries. However, the General German Automobile Club (ADAC) even recommends a minimum of 3 mm for summer tires and 4 mm for winter tires because lower tread depths lead to decreased traction and significantly increased stopping distances on wet or snowy road surfaces (Kroher, 2021). In order to maintain a safe car behavior, the Sebastian Huber et al. This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 United States License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
tire tread depth has to be continuously monitored. Usually, this inspection is conducted with manual gauges by the vehicle's driver, at the workshop or at technical inspection associations.
In the dawn of autonomous vehicles, the driver transitions to a mere passenger and thus considers himself less responsible for the vehicle's state of health. This aggravates with carsharing services, where each customer expects the vehicle to be in a safe condition and does not want to be bothered with monitoring activities. Tire inspection combined with regular vehicle service at the workshop may be sufficient for internal combustion engine vehicles (ICEV) that demand service intervals of about 30,000 km. Battery electric vehicles (BEV) have longer service intervals due to fewer moving parts in the motor and are thus more susceptible to worn tires (Tesla, 2022). Although vehicle owners in Germany are obliged to visit a technical inspection association for vehicle inspection every two years, driving high mileages per year can lead to late detection of critical tire wear. Furthermore, many countries worldwide do not have this requirement for regular vehicle inspection at all.
The above mentioned points motivate the need for an automated tread wear detection system. However, existing solution are either inaccurate or inconvenient. With TireEye we present an optical tire wear detection system that features four main advantages which are our main contributions for this paper: 1. The systems provides a mean absolute error of less than 0.6 mm, which enables valuable remaining useful life (RUL) predictions.
2. TireEye is an on-board measurement without the need for custom tires.
3. We display an adaptive Canny edge detection algorithm to compensate for varying lighting conditions. 4. Multiple additional use cases that may redeem the sensor costs (e.g. tire pressure estimation, tire damage detection, road friction coefficient estimation).
The remainder of this paper is structured as follows: After highlighting related work in Section 2, Section 3 presents the developed approach. A summary of the results is given in Section 4, followed by their discussion in Section 5. The paper concludes with a summary of the most important findings and an outlook for further research in Section 6.

RELATED WORK
Approaches for tire wear detection can be divided into methods based on series sensors and methods that require dedicated sensors additionally.

Tire Wear Detection Using Series Sensors
Signals from series sensors include wheel speed, vehicle body accelerations, and mileage. The most basic approach is assuming a constant wear across mileage. However, tread wear is strongly dependent on tire model (summer/winter, make, dimension), vehicle (driven axle, power, load), driving style, and environmental factors (road surface, temperature) (Continental, 2022). Thus, this approach may only be used as additional plausibility check for other methods. Steidel, Halfmann, Bäcker, and Gallrein (2016) take into account these variabilities by creating a catalog of load cases and their corresponding tire energy losses, which correlate with tread wear. Total tread wear can then be computed by partitioning all drives in segments of constant load and summarising across energy losses. Methodologically, this approach is load spectrum-based, a common practice in mechanical engineering use cases. However, the resulting accuracy is questionable since not all environmental factors can be accounted for. In addition, if the tire is changed, it must be manually registered to reset the tread wear counter.
An obvious approach to indirectly measure tread wear without additional sensors is exploiting the relationship for the tire's effective rolling radius R e with the longitudinal vehicle speed V x and the angular wheel speed ω. Equation 1 only holds true under the condition that there is no longitudinal wheel slip. R e is defined to be the tire's radius when rolling with no external torque applied and it decreases with progressing tread wear. Since the tire flattens in the contact patch, this value lies somewhere between the tire's undeformed radius and its static loaded radius. V x is measured using a global navigation satellite system (GNSS), whereas ω is provided by the electronic stability control (ESC) system. Miller, Youngberg, Millie, Schweizer, and Gerdes (2001), Ryan and Bevly (2012), and Lundquist, Karlsson, Ozkan, and Gustafsson (2014)-among others-exploit Equation 1 for tire radius estimation. However, the use case in focus of these publications is not tread wear but pressure loss detection. This indicates that the influence of tire pressure on R e is significantly bigger than tread wear. Additionally, R e depends on wheel load, tire design (radial steel-belted or bias ply), and vehicle speed (Pacejka, 2006). There may also be influences based on tire model and tire age, which would be even harder to account for.

Tire Wear Detection Using Dedicated Sensors
Regarding off-board sensors, Nevin and Daoud (2014) present a drive-over machine-vision sensor and compare its accuracy to traditional mechanical gauges. They emphasize the higher repeatability of the automated drive-over solution. Deviations of less than ±0.8 mm have been reported for both, the automated and the manual solution. Wang et al. (2019) even report an absolute error of less than 0.2 mm for their laser-based drive-over solution. Borgen, Mott, Newcamp, and Abrecht (2022) also use an optical sensor for tire condition and damage monitoring. Their Light Detection and Ranging (LIDAR) sensor achieves accuracies of ±1 mm in indoor environments and ±2 mm in outdoor environments. An increase in background infrared noise generated from the sun is mentioned to be the main cause of the decreased accuracies outdoors.
Tire manufacturers have been promoting intelligent tires for years. These tires are equipped with additional sensors to measure tread wear amongst other tire parameters. Of course, this leads to additional costs for the customer. There is no universal communication protocol between tire and vehicle yet. Consequently, customers cannot mount any intelligent tire but only those approved by the vehicle manufacturer. The manufacturers have given no exact information on the measurement errors of intelligent tires. Tyrata (2022) provides an internal tread sensor that they claim to be evolved from electric field reflection of carbon nanotube-based sensing electrodes. InWheelSense (TDK, 2022) is a retrofittable sensor setup for multiple purposes. It communicates with an external receiver via Bluetooth and features its own energy harvesting module for power generation from wheel movements. The built-in sensors include a 6-axis inertial measurement unit (IMU) and pressure and temperature sensors. Other potential use cases are tread wear measurement, road surface condition detection, and loose wheel indication. However, no further explanation on specific implementations is given. Lastly, Continental (2020) claims to be able to predict tire tread depth with sub-millimeter accuracy using a fusion of proprietary tire sensors and telemetry data. No further information on the sensors being used is given.
Prabhakara, Singh, Kumar, and Rowe (2020) use millimeter wave radar sensors placed in the tire wells to measure tire wear with an error of 0.68 mm. They exploit the tire's rotation with inverse synthetic aperture radar to enhance the range resolution. If road debris gets caught in the tire grooves, the perceived tread depth would be lower than the actual depth.
To compensate for this effect, they embed metallic strips in the grooves. These strips have a much higher reflectivity than debris and thus their reflections dominate. Additionally, their approach can locate harmful foreign objects like nails or screws.
Acoustic emissions could also be used as means to assess tire wear. Tong, Wang, Yang, and Wang (2013) report that tire wear is the main single tire-related influential factor on tire noise. Other factors include tire pressure and vertical load. Jha, Prasenjit, Karthikeyan, Madhav, and Mukhopadhyay (2019) investigate artificially worn tires on a test rig and find that sound pressure levels decrease with progressing wear. They point out that ageing effects have not been considered and may also influence acoustic emissions. Also, there is a high dependency on vehicle speed and pavement type (Masino, Foitzik, Frey, & Gauterin, 2017) that would need to be compensated for. No implementation of an audiobased tread wear detection system is known, yet.

METHODOLOGY
This Section presents the basic idea for tire tread wear detection as well as its specific implementation.

TireEye Setup
TireEye is a camera device mounted vertically inside of the wheel well ( Figure 1). There are no special requirements on the technical properties of the camera and in the scope of this paper, a smartphone camera (Samsung Galaxy A3 2017) has proven to be sufficient. To reduce computation times, the resolution has been set to 1920×1080. The position of the camera is chosen to be perpendicular to the road and tangential to the tire. This way, the outline of the longitudinal main tread groove can be detected since it provides a high contrast to the road surface ( Figure 2). The tread depth d is then measured as the distance perpendicular from the groove to the tread bar. Canny edge detection is being used to provide the groove's outline. Figure 2. Tread depth d as detected from the camera's perspective

Adaptive Canny Edge Detection
To detect the tire tread we apply Canny edge detection (Canny, 1986) which is widely used in computer vision applications. The algorithm typically consists of four stages: 1. Use Gaussian filter to smooth images and remove noise. 2. Compute intensity gradients for each pixel. 3. Thin multi-pixel wide ridges down to single pixel width (non-maximum suppression). 4. Apply upper and lower threshold for edges.
The upper and lower thresholds are tuneable parameters and control the algorithm's sensitivity. Intensities above the upper threshold are used to start an edge curve. If the neighboring pixels are above the lower threshold, the edge is continued. Intensities below the lower threshold are discarded as noise. Experiments have shown that in our case an additional median filter beforehand leads to better results.
Since the optimal thresholds depend on the lighting conditions, which are constantly changing, we developed an adaptive Canny edge detector that automatically parameterizes itself. To this end, an edge's intensity is defined as the ratio of edge pixels over the total number of pixels. The edge detection is repeated with varying thresholds until the edge intensity lies in a predefined range. Figure 3 shows the results for different edge intensities. With the configuration in Figure 3c, the outline of the tire is separable from the road and the tread groove can be further processed.

Tread Depth Measurement
After extracting the edges, the main tread groove has to be localized and measured. Since there are still many edges that correspond to dirt or irregularities on the road, all edges whose length is below a defined threshold are removed. To compensate for discontinuous lines due to mistakes, a tolerance is considered as to which extent two points may be apart and still form a line. Then, the tread groove is localized by taking advantage of its U-shaped form (Figure 2). A few heuristics are applied to achieve this: • The region of interest shows a high vertical extent. • It is adjoined by horizontal lines that constitute the tread groove and the tread bar. • It is symmetrical relative to a vertical axis.
After these constraints, the region with the highest vertical extent is chosen for further processing. Everything above the upper horizontal line and below the lower horizontal line is deleted. Finally, the vertical distance between the horizontal lines can be measured, which yields the tread depth d p in pixels.

Calibration
To obtain a tread depth in millimeters, a calibration of the measurement is required. Since the distance between camera and tire is constantly changing due to road excitations and vehicle load changes, a constant conversion factor between pixels and millimeters would not be feasible.
However, every tire has tread wear indicators (TWIs) with a height of h TWI = 1.6 mm to indicate the mandatory minimum tread depth (Figure 4). These TWIs are placed inside the main grooves and the radial distance between TWI and tread surface is the remaining useful tread.  (Continental, 2002) As the tire rotates, a TWI is regularly at a position where its shoulder lies directly in the camera's line of sight. This leads to a significantly lower perceived tread depth d p,min than in the adjacent frames. The TWI's height in pixels results in The actual tread depth d can then be computed as follows:

RESULTS
We define three setups which are used to evaluate the tread depth detection algorithm: I Manually turning the wheel on a stationary demonstrator (only rotational movement) II Manually pushing a rear axle demonstrator across different surfaces (rotational and lateral movement) III Full vehicle test in real world Table 1 shows three tires which have been used in these experiments. They all have different dimensions and are of different makes. However, each tire has only been used for one of the three setups. As a first outcome, the resulting tire tread measurement deviations are depicted in Figure 5.  Setup III is the most realistic, but also the most challenging scenario. Additional vibrations from the vehicle engine, dirt on the road and on the tires as well as varying lighting conditions lead to a MAE of 0.57 mm.

DISCUSSION
The results in Section 4 indicate that TireEye's accuracy is sufficient for tread wear detection. In contrast to stationary methods, the tire's whole circumference is measured so that unevenly worn tires cannot lead to false negatives. However, there are still challenges which need to be addressed.

Challenges
The challenges include • Debris on the camera: Dust and mud may accumulate on the lens and lead to blurry or black images. To reduce soiling through dirt hurled up from the tire, the camera is located at the front of the wheel. Also, protective plates might further reduce susceptibility to soiling. Active cleaning systems comparable to headlight washer systems could also be applied. If the system detects soil-ing of the camera lens, measurements can be flagged as invalid.
• Debris on the tire: Stones or snow stuck in the tread groove lead to a smaller perceived tread depth, while debris on the tread bar may imply too big depths. As tire tread wear is usually a slow process that stretches across several thousand kilometers, a step detection can be implemented so that wear gradients above a defined threshold are discarded. Normally, debris comes off after driving awhile and the correct tread depth can be measured again. Additionally, a debris detection using advanced computer vision methods, like convolutional neural networks, can be used to preventively discard measurements with soiled tires.
• Lighting: Although TireEye is resilient to varying lighting conditions due to adaptive Canny edge detection, it still relies on external illumination. Consequently, no measurement can be conducted at night. If this is an unacceptable constraint, an infrared light source needs to be added.
• Perspective: Although no special tires are required, they need to feature a longitudinal main groove. This does not apply to all tires and constitutes a major constraint of the measurement principle. Also, the perspective needs to be adjusted so that the groove in focus is centered. Adjacent grooves cannot be measured with the same accuracy.
• Production tolerances: The calibration of the measurements relies on the TWI's exact height of 1.6 mm (Section 3.4). Our investigations reveal an average error of 0.02 mm for the TWI, which corresponds to 1.25 %.
• Wheel speed: To measure the full height of the TWI, its shoulder has to be perpendicular to the camera's line of sight. This poses additional constraints on the minimum frame rate or the maximum wheel speed. Assuming a fixed frame rate of 30 fps, the wheel speed should not exceed 1 km h −1 . Since this range of speed is passed at each setting off, stopping, and parking event, the restriction can be sufficiently satisfied. The positive outliers in Figure 5 indicate that in the respective experiments the TWIs have not been captured in the instant of perpendicularity and thus the perceived h TWI,p is lower than their full height. According to Equation 3, this erroneously leads to a higher tread depth and may provide a false sense of security.
• Steering: If TireEye is applied at a steered axle, it can only be used when the steer angle is zero. Otherwise, the main tread groove would not be positioned as expected. This constraint is regularly satisfied, e.g., when stopping at a red light.
• Costs: Although cameras are nowadays mass products, automotive grade equipment is usually much more expensive. With the pricing pressure of the automotive industry, costs may be the biggest argument against mounting cameras inside of each wheel well. One alternative would be to only provide one camera per axle and assume consistent wear for the other side. Another option is to cover further use cases with these additional sensors to amortise them faster (Section 5.2). Table 2 shows how TireEye compares to other methods. Although it stands out in accuracy, issues regarding robustness and cost have to be addressed.

Additional Use Cases
TireEye has the potential for a multitude of additional use cases which will be presented in the following.
• Tire pressure estimation: Incorrect tire pressure may lead to blow-outs and reduced traction. The curvature of the tire's outline in the images enables to infer its pressure. If the outline is concave, the pressure is too low, whereas a convex outline corresponds to excess.
• Punctures: Foreign objects like nails or screws puncturing the tire pose a high accident risk due to potential sudden pressure loss. However, they could be detected with computer vision by using their shape and color.
• Incorrectly adjusted chassis: Misaligned camber and toe lead to unevenly worn tires. Even though the exact tread depth cannot be measured across the width of the tire, a high deviation between the two flanks could be detected.
• Tire model identification: Knowledge of the mounted tire model provides the benefit that its specific wear index can be used to check the measured tread depth for plausibility. Each tire features a model-individual tread pattern. This can be used like a fingerprint to identify make and model. Convolutional neural networks trained on a set of tire pictures are suitable for this purpose.
• Detection of worn rubber bearings: Worn rubber bearings impair vehicle stability due to additional wheel movements relative to the steering. This can be detected by correlating the steering wheel angle with the wheel's steer angle. The steer angle is derived from the tire's outline in the pictures.
• Side slip angle estimation: Side slip angle is an important factor for ESC systems. As the camera not only captures the tire, but also parts of the road, the vehicle's movement relative to the ground can be estimated. This basically constitutes a correvit sensor.
• Road friction coefficient estimation: Sudden changes in road friction due to ice or snow can be detected by color. As the camera is located in front of the wheel, there is even the possibility to pre-condition the ESC system. For smaller changes in road friction, the asphalt's granularity may be used as indicator.
• Animal detection: Autonomous cars have to be able to observe their surroundings before setting off. Otherwise, animals-like cats sleeping in front of the wheel-may be run over. TireEye is able to observe these areas and could prevent the car from starting.

CONCLUSION
This paper presents the concept of a novel method for camerabased tread wear detection. As the measurements are conducted on-board the vehicle, there is no need for inconvenient manual interventions. This increases safety not only for the passengers but also for other traffic participants. Tracing the tread depth across many drives enables extrapolating the tire's RUL. Fleet managers can then plan their workshop appointments accordingly and reduce vehicle downtime. Also, the spare part management of vehicle manufacturers can be optimized if the demand for new tires is known beforehand.
We are using three setups to evaluate the quality of this method and use a stationary demonstrator, a manually pushed demonstrator, and a full vehicle to conduct the experiments. The newly developed adaptive Canny edge detection enables to perform measurements resilient to varying lighting conditions. During the full vehicle test, a detection accuracy of 0.57 mm has been recorded. This beats state-of-the-art for on-board tire wear measurement. In the other two setups, the accuracy was even higher.
For TireEye to be ready for series application, a few challenges have still to be mastered-the biggest of which is component costs. In our future work we will cover additional use cases with TireEye to redeem these costs.