Micro Displacement and Strain Detection for Crack Prediction on Concrete Surface Using Optical Nondestructive Evaluation Methods

Continuous inspection of the concrete structures is important for extending the operating life span of nuclear power plants. Restricted physical accessibility to the nuclear plant structures, due to concerns of radiation, presents a unique challenge to the conventional visual inspection and contact-type nondestructive evaluation (NDE) technologies. Digital image correlation (DIC) is an optical NDE method that can measure the structural parameters such as displacement and strain. However, it is highly challenging to accurately detect micro displacement on the concrete surface because of weathering and change in illumination conditions. Usually, an artificial speckle pattern with good contrast to the object surface is needed for calibration and tracking, but it is difficult to apply in the field. In order to be able to detect micro surface strain for crack prediction in outdoor environment, a DIC-based NDE technology is developed to significantly improve the measurement accuracy by incorporating the orientation code matching (OCM) technique, a robust and accurate template matching algorithm. Concrete specimens were built and tested under four-point bending. Using the remotely measured images, the OCM incorporated DIC method successfully predicted concrete cracks before they emerged on the surface. The experiments also demonstrated the robustness of the method against the optical noise including weathering and change in illumination conditions.


INTRODUCTION
Built in the late 1970s, the U.S. fleet of 100 commercial nuclear power reactors is now middle-aged.There is an urgent need to develop technologies for extending their operating lifetimes beyond 60 years.Concrete structures are an important part of a light water reactor, proving foundation, support, and containment functions.Age-related deterioration in concrete is causing significant concerns about continued safe and efficient operation of these nuclear power plants.Currently safety assessment of the nuclear power plants (NPP's) across the nation, in which detection of concrete degradation is one of the most important assessment criteria, relies mainly on routine visual inspection.Several contact type NDE techniques based on expansion monitoring, acoustic waves, ultrasonic testing, impact echo, acoustic emission, and ground penetrating radar have been evaluated for inspection of concrete deteriorations (Giannini 2012).However, because of the structure's large size and limited accessibility in certain areas in a nuclear power plant, it is difficult to perform such inspections frequently and efficiently.But, there is also a need for continuous monitoring that would permit the evaluation of internal strain/stress/damage states in the bulk of the structure.
Non-contact NDE methods based on DIC can be applied for remote, continuous monitoring (or periodic inspection) of micro displacement on the concrete surface.Employing template matching, the conventional DIC methods have been used to detect displacement and strain of structural surface from continuously captured digital images in an indoor controlled environment (Hohmann, Bruck, Esselman, Yim & Schmidt, 2013).However, the conventional DIC methods suffer from measurement errors when applied in an outdoor environment where surface conditions might _____________________ Longxi Luo et al.This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 United States License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.change due to weathering, temperature/moisture effect and changing lighting conditions.In order to overcome this technical challenge, this study proposes a novel OCM incorporated DIC approach to significantly improve the measurement robustness against such environmental optical noise.The OCM technique was developed and its effectiveness was demonstrated through several outdoor field experiments for real-time, remote monitoring of bridge dynamic displacements in varying lighting conditions (Fukuda, Feng, Narita, Kaneko, & Tanaka, 2013) (Feng, Fukuda, Feng, & Mizuta, 2015) (Feng, & Feng, 2015).
In this project, specimens of reinforced concrete beams were fabricated in the laboratory.The beam specimens were then subjected to four-point bending to gradually introduce surface deformation.The surface displacement/strains were continuously monitored.The OCM incorporated DIC method and the conventional DIC methods were applied for crack prediction.A point wise finite element method inspired by point-wise least squares method were used for converting the surface displacements to strains.The OCM incorporated DIC method is compared with the DIC methods with regard to robustness against optical noise in simulated outdoor environment.

DIGITAL IMAGE CORRELATION AND OCM INCORPORATED DIC
Digital Image Correlation (DIC) has been extensively used for template matching.In the DIC method, a square reference subset in the reference image is correlated with a square deformed subset centered in a deformed image (Figure 1).Any reference subset or deformed subset is consisted of (2 + 1) × (2 + 1) pixels.Each pixel records a measured intensity.In the sum of squared difference method (SSD), the correlation value can be as simple as pixel-by-pixel summation of squared difference between the reference subset and deformed subset Eq. ( 1).
In the normalized sum of squared difference method (NSSD), the correlation value from SSD is normalized as shown in Eq. ( 2).The cross correlation method (CC) Eq. ( 3) and the normalized cross correlation method (NCC) Eq. ( 4) are also presented. (1) In Eq. (1) -Eq.(4), f(x, y) represents the grey level intensity at the coordinate (x, y) in the reference subset and g(x ′ , y ′ ) represents the grey level intensity at the coordinate (x ′ , y ′ ) in the deformed subset.Variables u, v are defined as the integers that represent the coordinate difference from the upper left corner point P(x, y) of a reference subset to the upper left corner point P′(x ′ , y ′ ) of the deformed subset: x ′ = x + u; y ′ = y + v (Figure 1).In SSD and NSSD, the best match between the reference subset and the deformed subset was represented by the minimum correlation value, while the bigger difference (less similarity) was represented by a higher correlation value.In the cross correlation method (CC) and the normalized cross correlation method (NCC), the best match between the reference subset and the deformed subset was represented by the maximum correlation value, while the bigger difference (less similarity) was represented by a lower correlation value.A correlation image can be created after correlating the reference subset with all possible deformed subsets in the deformed image.Each data in the correlation image is used to represent a correlation value.The displacement of a reference subset can be tracked by calculating the coordinate change from the reference subset to the best match deformed subset.
On the other hand, the OCM technique is based on matching gradient information around each pixel computed in the form of orientation codes, rather than the gray level intensity directly.It was shown by the study that the matching could be made robust by using gradient of the intensity values of adjacent pixels rather than the actual value of a pixel (Kaneko, Murase & Igarashi, 2002).In the OCM method, the intensity of each pixel of an grayscale image are replaced by orientation code that is obtained by quantizing the orientation angle at the corresponding pixel position in the grayscale image, as illustrated in Figure 2. The orientation angle θ i,j employed by OCM is determined by the quotient of gradient in y direction over gradient in x direction as in Eq. ( 5).In Eq. ( 5), I x and I y are the gradients at point P(i, j) of the image in x direction and y direction respectively.The orientation code is defined as the filtered and normalized orientation angle θ i,j .The steepest ascent orientation evaluated from the pixel neighborhoods was represented by the orientation angle.The orientation codes thus obtained contain gradient information and are a function of the texture and shape of the object, hence invariant to object translation and the effects of shading, background and illumination variations.More detailed explanation of the calculation algorithm of the OCM algorithm can be found in some original papers (Fukuda et al. 2013;Ullah et al. 2001).
In the conventional DIC methods a subset is tracked based on the intensity level of the pixels in the subset, while the OCM method transforms intensities of the grayscale images into orientation codes to form an OCM image that contains gradient information.Two grayscale images of the same object may have different intensities when they are taken under different environment conditions.For example, Figure 3 shows three images under different light exposures, but their OCM images are almost the same The intensities of a grayscale image may change in varying environment conditions (e.g.lighting condition or weathering), making it difficult for the conventional DIC methods to track the object.In contrast, the orientation codes in an OCM image will not be changed by variation in environmental condition because the texture and shapes of the objects do not change; therefore a subset can be tracked using the OCM method.Consequently, the OCM method is more robust than the conventional DIC methods when the object surfaces are subjected to change in environmental conditions.
In the OCM incorporated DIC method, the sum of squared difference method (SSD) (Eq.1) is used for template matching while the summation of error function was used in the original OCM method.The OCM incorporated DIC method inherits the robustness of the OCM method; therefore it is also insensitive to object translation and the effects of shading, background and illumination variations.

DISPLACEMENT/STRAIN MEASUREMENT ALGORITHMS
A displacement measurement method was applied for calculating displacements from results of template matching.
A sub-pixel resolution algorithm proposed by Gleason, Hunt and Jatko (1990) was employed to increase resolution of displacement measurement.A pointwise finite element method inspired by pointwise least squares method was used for more accurate strain calculation from displacement measurement.C++ program language (Stroustruph, 1983) and open source computer vision (openCV) library (Intel Corporation, Garage & Itseez 2000) were used for programming in this project.

Displacement Measurement and Sub-pixel Resolution
A displacement measurement method was used for calculating the displacements.In the beginning of displacement measurement, a region of interest (ROI) in the reference image is selected.Displacement will be measured in the ROI.Since only the relative displacement in ROI is needed for crack prediction, the total body displacement of the ROI is first tracked using template matching, and then subtracted in displacement measurement to remove the effect of camera movement.The ROI in the reference image is meshed into reference subsets with the same dimension (40 × 40 pixels).The displacement of each reference subset is tracked in the deformed image using the conventional DIC methods and the OCM incorporated DIC method.The tracking area is restricted to be 16 pixels larger than the reference subset in width and height to reduce the calculation time.A correlation image is obtained by correlating the reference subset with the deformed subsets located in all possible positions in the tracking area.Displacement measurement of a reference subset in pixel resolution is obtained by finding the maximum (CC and NCC algorithm) or minimum (SSD and NSSD) value in the correlation image.To obtain higher resolution, a sub-pixel resolution algorithm is needed.
To increase the resolution of the displacement measurement, a sub-pixel resolution algorithm proposed by Gleason et al. (1990) was employed.In the sub-pixel algorithm, it is assumed that the value distribution within a small window (3 × 3 pixels) of the correlation image is paraboloid.Least square method was used to calculate the coefficients of the paraboloid surface which fits through all the points in the window.By finding the peak value on the paraboloid surface, the extreme correlation value and its coordinate in the correlation image are obtained.The coordinates of the extreme correlation value M(x ′′ , y ′′ ) and the upper left corner point of the reference subset P(x, y) are compared for calculating the displacement of the reference subset.The displacement vector D(, ) is the coordinate change from P(x, y) to M(x ′′ , y ′′ ) : D(, ) = M(x ′′ , y ′′ ) − P(x, y) (Figure 4).Sub-pixel accuracy of the displacement measurement was achieved using this method.In order to eliminate boundary difference between each subset, the displacement measurement was smoothed using an averaging box filter.

Strain Calculation Using Point-Wise Finite Element Method
In this study, a pointwise finite element method is proposed, which assumes bi-cubic distribution in both  and  directions within each window in the displacement measurement image.Each window is consisted of a set of 16 (i.e. 4 × 4) points from the displacement measurement.
The displacement distribution within each window can be approximated by the bi-cubic shape functions as in Eq. ( 6).
The derivative vector (  ) is obtained by the matrix product of the derivative of the shape functions (  ) and the displacement vector (   ) (Eq. ( 7)).The bi-cubic shape functions ( 1  to  16  ) can be found in references of Finite Element Method (Onate 2009) (Fish & Belytschko 2007).The Green-Lagrange strain tensor can be calculated from the derivative vector (Eq.( 8)) (Geers, Borst & Brekelmans, 1995).In Eq. ( 8),  is the identity matrix and ( ) are the gradient in  direction and  direction of the displacement measurements respectively.Because the strain is only calculated at the center of the window which moves point by point, the accuracy of the strain results is improved.The pointwise moving method was inspired by the pointwise least squares method used for strain estimation by Pan, Asundi, Xie and Gao (2009).Because bi-cubic nonlinear displacement distribution is assumed in the pointwise finite element method, the strain results can be further improved.The adaptation of finite element method was inspired by the displacement interpolation method by Jone, Siberstein &White (2014).The flowchart of the displacement/strain measurement algorithm is displayed in Figure 5. where

Experiment Setup and Data Collection
The reinforced concrete beam specimens were fabricated by mixing Type I cement, water and #16 sand.For each specimen, approximately 900 gram sand, 300 gram cement and 120 gram water were used.The size of the beams is 14 × 2 × 1 inches.None artificial pattern was applied to the concrete surface.Reinforcing cages were built to reinforce the concrete specimens.The size of the reinforcing cage is 12.5 × 1.5 × 0.75 inches.The specimens were doubly reinforced in longitudinal direction.The #3 reinforcing bar was used for longitudinal reinforcement and Ø 1.6  steel bars were used for evenly distributed closed stirrups for lateral reinforcement.Figure 6  The RC beam specimens were subjected to four-point bending to gradually introduce surface deformation, using the Instron 5984 34k Universal Testing Machine.The test was controlled at the extension rate of 0.005in/min.
A digital camera with effective pixel count of 36.3-millionpixels ( 7360 × 4912 ) was used to capture the surface displacement at a six second interval.Eventually the concrete surface cracked under bending.Roughly 100 surface images were captured before cracking.The first surface image without environmental noise was used as the reference image while the others were used as deformed images.Among the deformation images, nine images were captured for each case in Type 2 condition.Nine images were captured for Type 3 condition.Five images were captured for each case in Type 4 condition.The camera was set up about one to two feet away from the specimen (Figure 6 (b)).For the future application, long focus lens will be used for capturing the surface displacement of a distant object.

Measurement without Optical Noise
In the condition without optical noise, collected image data were processed using the conventional DIC method (NCC), and the OCM incorporated DIC method for crack prediction.Figure 8 and Figure 9 respectively plot the concrete surface strain analyses results using the NCC and the OCM incorporated DIC method.Both show two curves indicating high strain concentration areas.Figure 10 and Figure 11 show the concrete surface displacement analyses results using NCC and OCM incorporated DIC method.In the displacement plots, one pixel is equivalent to 0.00142 inches (0.036mm).Both plots also show two curves indicating sudden displacement change.The curves obtained by either method match well with the actual locations and positions of the cracks formed on the post-cracking concrete surface as in Figure 7.The displacement and strain plots were obtained before any visible crack emerged on the concrete surface.The above results were obtained from analysis of the images taken on the rough concrete surface.
The OCM incorporated DIC method can also be applied to a smooth concrete surface.The strain distribution (E xx ) on the smooth concrete surface obtained by the OCM incorporated DIC method was plotted in Figure 13.Three curves can be found in the strain plot indicating high strain concentration areas.The surface displacement on the smooth concrete surface obtained by the OCM incorporated DIC method was plotted in Figure 14.Three curves can also be found in the displacement plot indicating sudden displacement change.The curves found in the strain and the displacement plots for the smooth concrete surface match well with the actual locations and positions of the crack formed on the postcracking concrete surface as shown in Figure 12.The cracks were highlighted in Figure 12 and Figure 7.The applicability of both the conventional DIC and the OCM incorporated DIC method for predicting the formation of cracks on concrete surfaces without optical noise was validated by the experiment results.

Measurement with Change in Illumination Condition
In an indoor controlled environment, both the DIC and the OCM incorporated DIC method can be used for crack prediction.However, it is highly challenging to make accurate measurement with the conventional DIC methods in outdoor environments, because of potential changes in surface conditions due to weathering and changes in lighting conditions.In the simulated environment, the conventional DIC methods produce a lot of errors in displacement estimation when the specimen is subjected to changes in environmental conditions.On the contrary, the robustness of the OCM incorporated DIC method is validated by results of the error distribution in displacement estimation.
The illumination condition on the surface was changed by partially blocking the light source (case T2-1) or reducing the brightness of light source (case T2-2) as displayed in Figure 15.The brightness in the shaded image is different from the brightness in the reference image.Errors were found in the displacement estimation obtained by the conventional DIC methods.Figure 16, Figure 17 and Figure 18 show the results of error distribution in displacement estimation obtained by the OCM incorporated DIC method, NCC, NSSD respectively.The rank of different methods from the best to the worst based on the amount of errors generated in the analysis is OCM incorporated DIC method, NCC, NSSD, SSD and CC.The comparison of the error area ratio of different methods was shown in Table 2.The error area ratio is defined as the ratio of the area with erroneous displacement measurements over the total calculated area; e.g., Ratio of red areas over the sum of the green areas and the red areas as shown in Figure 16, Figure 17 and Figure 18.The shapes of the error distribution found in Figure 18 are similar to the shapes of shading on the concrete surface in Figure 15.Since there are more errors in SSD and CC than NSSD, the plots of the error distribution obtained by SSD and CC are not shown.The OCM incorporated DIC method, compared to the conventional DIC methods, produces significantly less errors in displacement estimation.
Furthermore, the conventional DIC methods and the OCM incorporated DIC method were applied to all the collected deformed images in case T2-1 and those in case T2-2 for analysis of error distribution in displacement estimation.Nine deformed images were collected for case T2-1 and nine for case T2-2.The average of the error area ratio in each case was calculated for each of the methods, as shown in Table 3.It is obvious from the table that the OCM incorporated DIC method is more robust than the conventional DIC methods against change of illumination condition, because the error is significantly smaller.
From the definitions in Eq. ( 9) -Eq.( 11), NSSD, SSD and CC are highly sensitive to both the linear scale of illumination and the offset of brightness.In Eq. ( 9) -Eq.( 11),  represents the effect of linear scale of illumination and  represents the effect of offset of brightness.The NCC method is insensitive to the linear scale of illumination because it does not change the correlation value of NCC method as in Eq. ( 12) ( ≠ 0;  = 0).However, NCC is still sensitive to the offset of brightness because it changes the correlation value as in Eq. ( 12) ( = 0;  ≠ 0).The change of the correlation value due to the offset of brightness in NCC method is small, so the error produced in NCC is much less than other conventional DIC methods.The OCM incorporated DIC method is insensitive to the linear scale of illumination or the offset of brightness because of its nature.
The orientation code calculated in the OCM method contains the information of gradient of the grayscale images.
The gradients of the images only change at the edge of the shaded area where the brightness suddenly change from the reference image to a deformed image, but the effect is negligible because the affected area is insignificant compared to the subset.The variation of lighting condition, such as the linear scale of illumination and the offset of brightness, is common in the outdoor environment.
Because of the significant errors in the displacement estimation by the conventional DIC methods, these methods cannot be used for crack prediction in outdoor environment.
On the contrary, the OCM incorporated DIC method can provide robust surface displacement estimation results for crack prediction in outdoor environment with changes in illumination condition.4. Figure 20, Figure 21, Figure 22 and Figure 23 show the results of error distribution in displacement estimation obtained by the OCM incorporated DIC method, NCC, NSSD and SSD respectively.A large amount of errors were found in the displacement estimation obtained by NSSD, SSD and CC.The shape of error distribution found in Figure 22 and Figure 23 is similar to the shape of water mark on the concrete surface as shown in Figure 19.Fewer errors were found in the result obtained by NCC.The least amount of errors was found in the result obtained by the OCM incorporated DIC method.
Furthermore, nine deformed images captured under simulated weathering were analyzed in terms of error distibution in displacement estimation using the conventional DIC methods and the OCM incorporated DIC method.The average of the error area ratio of the deformed images obtained by each method was calculated for comparison as displayed in Table 5.From the table, it was found that the OCM incorporated DIC method is more robust than the conventional DIC methods against simulated weathering.The influence by the water on the brightness change on the concrete surface is similar to the effect by the combination of the offset of brightness and the linear scale of illumination.The OCM incorporated DIC method is insensitive to offset of brightness or linear scale of illumination.
Weathering of the concrete structures in outdoor environment is inevitable.When SSD, NSSD, CC and NCC are used for crack prediction in outdoor environment, a large amount of errors will be generated in the measurement results.The OCM incorporated DIC method effectively reduces the measurement errors therefore provides a robust method for crack prediction in outdoor environment.In the outdoor environment, structures are always exposed to both change in illumination condition and weathering.
The surface condition was changed by partially blocking the light source and spraying water on the surface of the specimen (case T4-1) (Figure 24) or reducing the brightness of light source and spraying water on the surface of the specimen (case T4-2).Only plots from case T4-1 will be demonstrated.Errors were found in the displacement estimation obtained by the conventional DIC methods.The comparison of the error area ratio of different methods was shown in Table 6. Figure 25, Figure 26, Figure 27 and Figure 28 show the results of error distribution in displacement estimation obtained by the OCM incorporated DIC method, NCC, NSSD and SSD respectively.A large amount of errors were found in the displacement estimation obtained by NSSD, SSD and CC.The shape of error distribution found in Figure 27 and Figure 28 is similar to the shape of the shaded area or the water mark on the specimen surface as shown in Figure 24.Fewer errors were found in the result obtained by NCC.The least amount of errors was found in the result obtained by the OCM incorporated DIC method.
Furthermore, each of the methods was applied to the five deformed images collected for case T4-1 and the five deformed images for case T4-2 for analysis of error distribution.The average of the error area ratio in each case was calculated and tabulated in Table 7. Similar to the previous cases, the OCM incorporated DIC method is more robust than the conventional DIC methods against the combination of change in illumination condition and simulated weathering.The effect of the combination of change in illumination condition and simulated weathering on the brightness on the concrete surface is similar to the effect of the combination of the offset of brightness and the linear scale of illumination.
The CC, SSD and NSSD methods are sensitive to change in illumination condition or raining condition because both the linear scale of illumination and the offset of brightness can change the correlation value in these methods.NCC is sensitive to the offset of brightness.The OCM incorporated DIC method is not sensitive to either the linear scale of illumination or the offset of brightness because the orientation angle θ i,j employed by the OCM method is determined by the quotient of gradient in y direction over gradient in x direction Eq. ( 5).
Because the linear scale of the illumination scales the gradient by the same magnitude in both directions, the quotient of the gradients, the orientation angle θ i,j and the orientation code remain the same as shown in Eq. ( 13).
Because the offset of brightness will not change the gradient in either direction of the image as shown in Eq. ( 14) and Eq. ( 15), the orientation angle θ i,j and the orientation code also remain the same.

CONCLUSIONS
Inspection of the concrete structures in nuclear power plants, where access may be restricted due to concerns of radiation, presents a unique challenge to the conventional visual inspection and contact-type NDE technologies.The proposed OCM incorporated DIC method enables prediction of cracks on concrete surface from remotely measured digital images.
In this study, concrete beam specimens were fabricated and tested under four point bending to gradually introduce deformation on the concrete surface.The conventional DIC methods and the OCM incorporated DIC method were applied to images of surface deformation captured before cracking.The strain/displacement measured by these methods successfully predicted the cracks before they emerged on the surface.Error distribution in displacement measurements of the concrete surfaces was used to compare the robustness of the OCM incorporated DIC method and the conventional DIC methods against simulated outdoor environment including change in illumination condition, simulated weathering and the combination of change in illumination condition and simulated weathering.Under these conditions, the OCM incorporated DIC method produces significantly less error in displacement estimation than the conventional DIC methods therfore provide a robust method for crack prediction in outdoor environment.

Figure 1
Figure 1 Schematic figure of the deformation model

Figure
Figure 4 Displacement calculation

Figure 5
Figure 5 Flow chart of displacement/strain measurement algorithm4.EXPERIMENTAL VERIFICATIONThe application of the DIC and the OCM incorporated DIC method described in the previous sections is to predict cracking on concrete surface.Experiment results of a specimen in four types of environment conditions were analyzed.In an environment without optical noise (Type 1 condition), the displacement/strain measurement obtained by the OCM incorporated DIC method and the conventional DIC methods were used for crack prediction.The OCM incorporated DIC method is then compared with different conventional DIC methods (CC, NCC, SSD, NSSD) in terms of robustness against optical noise in simulated conditions including changes in illumination (type 2 condition), weathering (type 3 condition) and the combination of change in illumination and weathering (type 4 condition).The change in illumination condition is simulated by blocking the light source or changing the brightness of the light source.The weathering condition is simulated by spraying water on the surface.Table1summarizes the tested environmental conditions.
(a)  shows the size and layout of the reinforcing cage and Figure6(b) shows the experiment setup.
Figure 6 Experiment setup

Figure 7 Figure 8 )Figure 9 )Figure 10 Figure 11 Figure 13 Figure 14
Figure 7 First two cracks formed on the concrete surface (rough surface)

Figure 16 Figure 17
Figure 15 Shaded concrete surfaces

Figure 19 Figure 22
Figure 19 Concrete surfaces with water mark in illumination condition and weathering of the concrete structures in outdoor environment is inevitable.Because of the significant errors in the displacement estimation by the conventional DIC methods, they cannot be used for crack prediction in outdoor environment.The OCM incorporated DIC method, on the other hand, effectively reduces the measurement errors and thus can be applied for crack prediction in outdoor environment.

Figure 24
Figure 24 Concrete surfaces with shaded area and water mark (case T4-1) Table 1 summarizes the tested environmental conditions.

Table 6
Comparison of error area ratio due to combination of change in illumination condition and simulated weathering

Table 7
Average value of error area ratio due to combination of change in illumination condition and simulated weathering in five deformed images