Super-resolving material microstructure image via deep learning for microstructure characterization and mechanical behavior analysis

Abstract

The digitized format of microstructures, or digital microstructures, plays a crucial role in modern-day materials research. Unfortunately, the acquisition of digital microstructures through experimental means can be unsuccessful in delivering sufficient resolution that is necessary to capture all relevant geometric features of the microstructures. The resolution-sensitive microstructural features overlooked due to insufficient resolution may limit one’s ability to conduct a thorough microstructure characterization and material behavior analysis such as mechanical analysis based on numerical modeling. Here, a highly efficient super-resolution imaging based on deep learning is developed using a deep super-resolution residual network to super-resolved low-resolution (LR) microstructure data for microstructure characterization and finite element (FE) mechanical analysis. Microstructure characterization and FE model based mechanical analysis using the super-resolved microstructure data not only proved to be as accurate as those based on high-resolution (HR) data but also provided insights on local microstructural features such as grain boundary normal and local stress distribution, which can be only partially considered or entirely disregarded in LR data-based analysis.

Introduction

Accurate representation of digital microstructure (DM) plays a pivotal role in modern-day materials research as the DM encompasses an extensive amount of data valuable for applications ranging from microstructure visualization and characterization to microstructure-based numerical modeling. To date, an extensive amount of effort has been put forth in developing tools and methods to take full advantage of DMs^{1,2,3,4,5,6,7,8}. Optical microscopy (OM), scanning electron microscopy (SEM), and electron backscatter diffraction (EBSD) techniques have been textbook approaches to visualize and characterize DMs for decades. Furthermore, 3-dimensional (3D) visualization and characterization of DMs using EBSD combined with a focused ion beam (FIB)¹ or X-ray tomography^2,3 have nowadays been popularly used in the field of advanced structural materials. Likewise, in the modeling sector, DMs are used to predict or analyze a wide variety of microstructure-sensitive properties such as strength and ductility⁴, thermal properties⁵, corrosion⁶, recrystallization⁷, and phase transformation⁸.

Data fidelity of DMs is inevitably compromised upon the acquisition or processing of DM data. A recent work by Bao et al.⁹ defined and quantified the missing information of DMs. The work stated that there are various sources that contribute to the loss of information, and some of these sources include instrumental resolution, technical specifications, and distribution of microstructural features⁹. One of the most commonly confronted problems due to information loss in dealing with DM data such as EBSD data is the poor resolution issues. Because the properties of a material heavily depend on the geometric details of its microstructure, limited spatial resolution may be critical in studying the mechanical behavior of the material. In particular, microstructure-based models that fully exploit DMs can suffer gravely depending on how the material properties of interest depend on the information lost. For instance, the damage behavior of high-strength steels with complex microstructures is significantly governed by stress localization and local phase morphology^2,10, meaning that an accurate and precise representation of local morphological features is critical in understanding the damage evolution of the high strength steels. Consequently, restoration or regeneration of information lost during the acquisition or processing of the DMs can be vastly valuable for accurate and comprehensive materials analysis. A possible solution to address the poor resolution issue in a DM is by enhancing the resolution of the DM image via super-resolution (SR) imaging. Conventional methods for SR imaging include upsampling low-resolution (LR) images through interpolation techniques such as the nearest neighbor, bicubic, or bilinear interpolations. Advanced SR techniques include reconstructing high-resolution (HR) microstructure from existing HR 2D distance correlation functions (DCFs) using algorithms such as cellular automata¹¹, Markov random field¹², and two-point exchange¹³. However, these methods not only require DCFs but also employ an iterative approach in upsampling LR images, which may be computationally exhaustive and time-consuming.

Concurrent with the rise of data-driven analysis in the past decade, amazing strides in SR imaging have been accomplished through deep learning algorithms such as those based on convolutional neural network (CNN)^{14,15,16,17,18}. In particular, Hagita et al.¹⁷ and Haan et al.¹⁸ have successfully utilized deep learning to upscale greyscale SEM images. These works demonstrated that regeneration of data fidelity in terms of image resolution is feasible with a deep learning approach within the structural materials landscape. These works mostly deal with SEM images, which are grayscale image data. However, there are other DMs that are in demand of SR imaging, namely EBSD image data, which contains more information than SEM images and is prone to produce jagged feature edges and oversimplification of features upon resolution loss. Moreover, most works dealing with SR of structural materials pay little attention to how SR can be effective in both microstructure characterization and numerical material behavior analysis.

In this work, we present a fast and accurate deep learning-based SR technique called SR residual network (SRResNet) to super-resolved LR EBSD image data. While we utilize synthetic microstructure images rather than actual microstructure images with the purpose of the network training to overcome insufficient data problems, our network is evaluated using both actual and synthetic microstructure images. We validate the super-resolving performance of the network through conventional image similarity measurement and microstructure characterization. We further demonstrate the effectiveness of super-resolving EBSD microstructure images by conducting finite element (FE) mechanical analysis on a super-resolved EBSD microstructure image. The results show that the developed SRResNet enables rapid acquisition of super-resolved EBSD images that are comparable to HR EBSD images based on image similarity metrics, microstructure characterization, and mechanical analysis using microstructure-based FE simulations.

Results and discussion

SR network

Residual neural network (ResNet)¹⁹, a specially structured convolutional neural network for allowing the training of deeper networks, is widely known to possess a powerful representation ability and achieves superior performance compared to conventional approaches in ill-posed problems such as SR and deblurring. Thus, ResNet approach has been considered in this study for SR (SRResNet)¹⁶ to enhance the resolution of DM images, which are inverse pole figure (IPF), Euler, and phase map images (Fig. 1a). The HR images are downsampled using nearest-neighbor interpolation as shown in Fig. 1b. The network performs to upsample LR microstructure images as input data into HR microstructure images as output data, with upsampling factors such as 4×, 8×, and 16x× defined by the network architecture shown in Fig. 1c. In detail, the network is composed of two modules: (i) a residual block that consists of convolutional layers, ReLu activation function, a convolutional layer, and skip-connection. (ii) an upsampling layer that uses repetition modules consisting of convolutional layers, pixel shuffle, and ReLu activation function according to the targeted upsampling factor, where pixel shuffle is to rearrange feature maps of $H \times W \times C \cdot r^2$ shape to form $rH \times rW \times C$ shape²⁰. The network’s input data, LR image patches of orientation and phase map, pass through the convolutional layer, ReLu activation function, 16 residual blocks, and convolutional layer in turn. Then, skip-connection is used to connect the output feature maps of the last convolutional layer and the input feature maps of the residual blocks. The resulting feature maps are passed to the upsampling layer, which is either 4×, 8×, or 16× upsampling module, followed by a convolutional layer. Finally, the output layer generates a super-resolved microstructure image with a sigmoid activation function. We build the networks for 16×, 8×, and 4× upsampling, respectively, which only differ in the upsampling module. For example, each network generates a $1000 \times 1000$ image, from a $64 \times 64$, $125 \times 125,$ and $250 \times 250$ LR images, respectively. The parameters, such as kernel size, channel size, pad, and stride, are shown in Table 1.

Fig. 1: Overall scheme in data acquisition and super-resolution neural network.

a Synthetic HR microstructure images of both set of IPF and phase images and set of orientation and phase images, b downsampling process of an HR image, and c the architecture of super-resolution network.

Table 1 The parameters of our network.

We train these networks using the Adam optimization algorithm²¹ with a learning rate set to 0.0005 during 50,000 iterations. Mini-batch size is set to 40, and the number of input image channels is set to 4 (3 orientation channels and 1 phase channel). Xavier initializer is used as the kernel initializer, and ReLu²² is used as the non-linear activation function. We use the mean absolute error (MAE) as a loss function instead of the mean squared error (MSE) used in the original SRResNet. The reason is that MSE encourages a network to find pixel averages of plausible solutions, leading to excessively smooth images to be restored, while MAE reduces the average error and results in sharper images being generated. The networks are trained on an NVIDIA TITAN Xp GPU and implemented with Tensorflow. Other training details are shown in Table 2.

Table 2 Hyper-parameters used for our network.

Image similarity measurement

We demonstrate that our SRResNet can transform LR microstructure images into HR images with remarkable performance in terms of image similarity metrics that are commonly used in the field of computer vision. We compare the results of our SRResNet with the results of conventional methods (bilinear, bicubic and nearest neighbor) in 4×, 8×, and 16× upsampling cases. We conduct SR experiments on sets of IPF and phase images and sets of orientation and phase images. For this, we first collect LR and HR microstructure images. The HR images are produced using as shown in Fig. 1a, where three Euler angles that represent an orientation are converted into RGB values by multiplying Euler angles by 255/360. The LR images are synthetically generated using the nearest neighbor interpolation, as shown in Fig. 1b. The nearest neighbor downsampling method synthetically generates LR images having sharp and discrete edges similar to actual LR EBSD images. We note that applying an anti-aliasing filter to prevent aliasing effects during the downsampling process is not suitable for EBSD images with sharp and discrete edges. The reason is that an anti-aliasing filter transforms jagged edges, which are high-frequency components and an intrinsic characteristic of EBSD images, into blurred edges, which are low-frequency components and rarely observed in EBSD images. Once our SRResNet is trained on the pairs of LR and HR images, the network can improve the resolution of any size of LR microstructure images. Figure 2 shows the qualitative result of 4×, 8×, and 16× upsampling of the LR test images that are 10% of the total collected 12,000 images. The first column shows the input LR images, the second column shows the HR images (ground-truth), the third, fourth, fifth, and sixth columns are the super-resolved images of the LR images using bilinear, bicubic, nearest-neighbor interpolation, and the SRResNet results, respectively. The rows represent results of 4×, 8×, and 16× upsampling for each of orientation and phase images. Although Fig. 2 includes only qualitative results for orientation and phase images due to a lack of paper margins, the same tendency is observed for IPF and phase images. One can see that results of our method are qualitatively more similar to the ground-truth, and shows clear boundaries in 4× upsampled case, compared to the bilinear, bicubic, and nearest-neighbor interpolation. For 8× and 16× cases, our SR images slightly differ from the ground-truth, but still, have the most plausible morphology. We also compare our SRResNet quantitatively with conventional methods using two image similarity metrics: peak signal to noise ratio (PSNR) and structural similarity index measure (SSIM), where larger values correspond to the higher similarity between super-resolved image and ground-truth. The PSNR metric, which calculates the average error for all pixels, may overlook reconstruction errors caused by grain boundaries since the errors are averaged with those caused by an in-grain area that occupies almost all pixels within an EBSD image. Considering that reconstruction error within EBSD images is mostly caused by grain boundaries than in-grain area, the SSIM metric, which measures structural similarity, is more suitable than the PSNR metric as a quantitative metric for EBSD image restoration. Figure 3 shows the quantitative results on all of the test sets of IPF and phase images and the test sets of orientation and phase images by an average of their results in terms of PSNR and SSIM. In Fig. 3a, b, for the 4× upsampling case, where 1000 × 1000 microstructure images are produced from $250 \times 250$ microstructure images, it can be observed that the SR results from the proposed method are very close to the ground-truth in terms of SSIM. It also outperforms conventional approaches in terms of PSNR. For an 8x upsampling case, where 1000 × 1000 microstructure images are produced from $125 \times 125$ microstructure images, the SSIM value of our network is approximately 0.95, which is similar to the SSIM value of conventional methods in a 4× upsampling case. Finally, $64 \times 64$ microstructure images are transformed into 1000 × 1000 microstructure images for the 16× upsampling case. Although its quantitative and qualitative performance is significantly degraded compared to 4× and 8× cases due to extreme loss of information in the LR input images, our networks perform better than other methods in terms of both SSIM and PSNR. We also evaluate the performance of the network using IPF maps obtained from EBSD data that are used in^23,24,25,26 and IPF map images directly captured from^6,27,28 are shown in Fig. 3c, d. We preprocess the data obtained from^6,27,28 using total variation (TV) denoising filter²⁹ and high pass filter for image sharpening since the SRResNet is trained with a sharp and noise-free image. Also, the IPF images are cropped to remove scale bars and legends. One can see that the overall performance decreased when published data are used instead of synthesized test data. This is expected since the test data obtained from published works not only include very strong in-grain misorientation but also contain noise and blur that cannot be completed addressed with a TV denoising filter and sharpening filter. Nevertheless, the SRResNet still outperforms other conventional upsampling techniques.

Fig. 2: Result of 4×, 8×, and 16× upsampling of LR test images.

Comparison among input LR, ground truth (HR), bilinear upsampled, bicubic upsampled, nearest neighbor upsampled, and SRResNet upsampled images for qualitative assessment.

Fig. 3: Performance different upsampling methods in terms of PSNR and SSIM.

Quantitative result of 4×, 8×, and 16× upsampling of the LR test images in terms of a PSNR and b SSIM for synthesized test data, and c PSNR and d SSIM for literature IPF map data^{6,23,24,25,26,27,28}. The boxplot depicts the first quartile (Q1), median, and third quartile (Q3), as well as lines showing a 1.5 interquartile range.

Impact of SR on microstructure characterization

Microstructure images obtained using EBSD techniques can be characterized by sharp and discrete edges. An exemplary HR microstructure image and its nearest neighbor interpolated LR counterparts are shown in Fig. 4. One can see that with the decreasing resolution, grain boundaries and phase interfaces become increasingly jagged. For polycrystalline materials, the average grain size and morphological details of each grain are pertinent to the performance of materials. Therefore, the influence of resolution and the effect of SR in characterizing the average grain size and morphological details of each grain is studied in this section.

Fig. 4: Synthetic inverse pole figure (IPF) and phase map images.

a High resolution (1000 × 1000) microstructure IPF image and its b low resolution (125 × 125) counterpart with phase maps as inset figures.

Microstructure characterizations are conducted with synthetic HR microstructure image data, which consist of 1000 × 1000 pixels, its downsampled LR microstructure image data, which consist of either $250 \times 250$ (4× case) or $125 \times 125$ (8× case) pixels, and super-resolved microstructure image data, which are reconstructed solely from either one of the LR microstructure images. A Matlab toolbox MTEX was used for characterization³⁰. Area fraction-based average grain sizes of HR, LR, and super-resolved microstructures are calculated using their respective IPF and Euler map images. Microstructural features are segmented based on the RGB values of microstructure images. It should be noted that the SRResNet may result in a very small change in the RGB values of some grains in the input LR image. This change is generally negligible, but an IPF image that has neighboring grains with very small color differences may require a lower segmentation criterion for grain segmentation. Assuming that a pixel in HR microstructure images corresponds to one micrometer, we used the mean relative error defined in Eq. (1) to quantify the accuracy in utilizing LR and super-resolved images to calculate average grain sizes. The mean relative error is defined as

$$\frac{1}{N}\mathop {\sum}\limits_i^N {\frac{{\left| {d_{i,{\mathrm{ground}}\,{\mathrm{truth}}} - d_{i,{\mathrm{test}}}} \right|}}{{d_{i,\,{\mathrm{ground}}\,{\mathrm{ruth}}}}},}$$

(1)

where $d_{i,{\mathrm{ground}}\,{\mathrm{truth}}}$, $d_{i,{\mathrm{test}}}$, and N represent the average grain size calculated from ith HR microstructure image, the average grain size calculated from LR or super-resolved microstructure image that corresponds to the ith HR microstructure image, and the number of microstructure images used to evaluate the mean relative error, respectively. The tested images had average grain sizes less than 100 μm to avoid images that only capture a few grains. The mean relative error of average grain sizes calculated using 250×250 LR images was 0.014. Similarly, the mean relative error in average grain sizes calculated using images super-resolved from the LR images was 0.012. Both average grain size calculations were relatively accurate. The mean relative error of average grain sizes calculated using 125 × 125 LR images was 0.045, which is 3.2 times larger than those based on 250 × 250 LR images. On the other hand, the mean relative error of average grain sizes calculated using images super-resolved from the 125 × 125 LR images was 0.016. That is, grain size calculation errors stemming from insufficient image resolution can be partly resolved by super-resolving LR microstructure images via SRResNet.

Local geometries of grain and phase boundaries may govern the localized behavior of a material. The effects of interfaces may initially be localized, but it is known that these effects substantially influence the final structural performance of materials. For example, grain and phase boundaries are known to act as a preferential site for void nucleation and growth. Void nucleation due to interface decohesion is strongly influenced by interface normal stress and accumulated plastic strain³¹. Hence, in a continuum framework, damage initiation at an interface heavily relies on the direction of applied stress with respect to the interface normal^31,32. For images with pixel data, interface normal can be estimated using the Sobel operator. The Sobel operator is a differential operator kernel that is popularly used for edge detection. By applying convolution between the Sobel operator and an image, one can estimate local derivatives of the image. Two Sobel operators for x- and y-directions are used to quantify the interface direction of any given microstructure. Given a point of convolution, the grain to which the point belongs is assigned a unit value and all other grains are assigned a value of zero. The resulting gradient map of one-fourth portion of an HR microstructure and its respective LR and super-resolved microstructures are shown in Fig. 5. Naturally, one can see that grain boundaries are very thick and jagged for the LR microstructure because the size of each pixel relative to the image it belongs to is large. For small features, most or entire pixels will belong to their respective boundaries. This may have important ramifications in properly assessing localized mechanical behaviors such as in-grain deformation heterogeneity and stress localization near boundaries. The thick boundary issue can be partly addressed by upsampling the image size using conventional nearest-neighbor interpolation. This conventional technique is very fast and simple. The nearest neighbor interpolation upsampling is simply splitting each pixel into multiple pixels with the same RGB values. As shown in Fig. 5d, the grain boundary thickness of the nearest neighbor upsampled microstructure is comparable to its respective HR and super-resolved microstructures. However, because conventional image upsampling techniques do not properly deal with the jagged edges, the grain boundary normal of the nearest neighbor upsampled microstructure is notably biased toward either x- or y-direction. Furthermore, other conventional upsampling techniques such as bilinear or bicubic interpolation upsampling yield very blurry images, resulting in extreme difficulty in properly defining grain boundaries. On the other hand, the grain boundary normal of the SRResNet super-resolve microstructure strongly resembles that of the HR microstructure.

Fig. 5: Normalized gradient map of microstructure images.

Normalized x-gradient (top), y-gradient (middle), and gradient magnitude (bottom) maps of a high-resolution, b low-resolution (4x), c SRResNet super-resolved, and d nearest neighbor upsampled microstructure images.

The benefits of applying SRResNet for microstructure characterization not only reside in accurate characterization of microstructural features but also in rapid acquisition of HR microstructure data. That is, in a fraction of a second, the super-resolved EBSD image data successfully regenerated a wealth of microstructural details, which was seemingly lost in the LR image data. LR EBSD scan typically takes less than 5 min, meaning that obtaining 4× or 8× upsampled HR EBSD data can take up to 80 or 320 min. Using the developed SRResNet in an NVIDIA TITAN Xp GPU environment, one can super-resolve LR EBSD data into 4×, 8×, or 16× upsampled data in only 0.11, 0.08, or 0.07 s, respectively. This provides an extremely time-efficient deep learning-based solution to obtain HR EBSD data. This high-speed acquisition of HR EBSD data is immensely valuable in amassing EBSD data, obtaining large area EBSD scans, and restoring information lost due to instrumental resolution limit.

Impact of SR on mechanical analyses

Small variations in microstructural features of material can substantially affect the mechanical behavior of the material. Mechanical modeling has been frequently utilized to understand how microstructural changes in a material can affect the mechanical behavior of the material. Among existing methods of modeling, the microstructure-based FE method has been a prevalent choice in studying the effect of microstructure on the mechanical behavior of advanced structural materials. Microstructure based FE method requires DMs as input, meaning that resolution issues associated with the input DMs may influence the simulated mechanical behavior of the DMs. Accordingly, information loss in DMs may limit the extent of FE-based mechanical analysis of the DMs.

To identify the impact of resolution on FE-based mechanical analysis, microstructure-based FE simulations using HR, LR, and super-resolved microstructures of a dual-phase HEA used in²³ are conducted. The LR microstructure data is created by nearest neighbor downsampling of HR microstructure data, and the SR microstructure data is recovered from the LR data using our SRResNet trained on synthetic microstructure images. Figure 6 shows the stress-strain curves of high entropy alloy using HR, LR, and super-resolved microstructures. The image resolutions are $240 \times 712$ pixels for the HR and the super-resolved microstructure images and $60 \times 178$ pixels for the LR microstructure image. One can see that the overall flow stresses for HR and super-resolved microstructures are almost identical. The flow stress for LR microstructure is slightly higher than those for HR and super-resolved microstructures. This is because more intense strain localization occurred in HR and super-resolved microstructures than in the LR microstructure due to insufficient mesh resolution in the model using the LR microstructure.

Fig. 6: Experimentally obtained and simulated stress–strain curves of a high entropy alloy.

The curves are obtained from the literature (square dot)²³, high-resolution microstructure data (line with an upward triangle), super-resolved microstructure data (line with a downward triangle), and low-resolution microstructure data (line with circle). The microstructure images below are Euler and phase maps used for finite element simulations.

Stress distributions of HR, LR, and super-resolved microstructures after 5% tension are shown in Fig. 7. In this case, HR and super-resolved microstructures exhibit similar stress distribution characterized by stress localized near the peripheral of hard secondary phase constituents. Meanwhile, in the case of LR microstructure, stress distribution seems to be less localized for small secondary phase constituents because LR microstructure does not retain sufficient resolution to adequately capture stress distribution within the small secondary phase constituents. This can also mean that average stress values for very small secondary phase particles calculated using the LR microstructure will be high compared with those calculated using either HR and super-resolved microstructures, which may partly contribute to higher flow stress for LR microstructure compared to those for HR and super-resolved microstructures. The discrepancy in stress localization behavior between HR and LR microstructures can be addressed by upsampling the LR microstructure image using nearest-neighbor interpolation. One can see that the nearest neighbor upsampled microstructure exhibits stress localized near the peripheral of the secondary phase constituents (Fig. 7). However, typical upsampling techniques such as nearest neighbor are limited in their ability to restore morphological details of a microstructure (Fig. 2), which heavily impacts one’s ability to conduct a thorough mechanical analysis that relies on such details. For any dual phase microstructures, normal stress acting on interfaces between soft and hard phases plays a vital role in damage initiation during deformation^31,32. One can calculate the normal stress acting on either side of the interfaces using FE simulated stress tensor and Sobel operator-defined interface normal. Hence, the normal stress acting on the hard phase side of the interface is calculated by using the stress tensor of all elements belonging to the hard phase that is adjacent to an element belonging to the soft phase and the interface normal of the hard phase element defined by the Sobel operator. Figure 8 represents the distribution of normalized normal stress from the hard phase region after 1 and 5% tensile elongation using HR, nearest neighbor upsampled, and super-resolved microstructures. The mean normal stresses were 262, 257, and 246 MPa for HR, super-resolved, and nearest neighbor upsampled microstructures, respectively, for the 1% tension case. The mean normal stresses were 347, 343, and 317 MPa for HR, super-resolved, and nearest neighbor upsampled microstructures, respectively, for the 5% tension case. One can clearly see that the distribution obtained using super-resolved microstructure strongly resembles the distribution obtained using the HR microstructure. Conversely, the normal stress distribution obtained using nearest neighbor upsampled microstructure resembles a bimodal type distribution, which is different from the normal distributions obtained using other microstructures. The bimodal distribution is observed because the interface normal of the nearest neighbor upsampled microstructure is biased toward either x- or y-direction.

Fig. 7: Simulated Mises stress distribution of the outlined region after 5% tension.

From left to right are the stress distribution obtained using high-resolution, super-resolved, low-resolution, and nearest neighbor upsampled microstructure data.

Fig. 8: Distribution of normalized normal stress from the hard phase side of the interface after 1 and 5% tensile elongation.

From left to right are the distribution obtained using high resolution, super-resolved, and nearest neighbor upsampled microstructure data.

Super-resolving experimentally obtained LR EBSD data

To verify the effectiveness of the SRResNet in super-resolving experimentally acquired LR EBSD image data, three sets of experimentally acquired LR and HR EBSD data were utilized. The first set of HR and LR EBSD data are CoCrFeMnNi high-entropy alloy (HEA) used in²⁴. The HR and LR EBSD were processed to have square grids with step sizes of 300 nm and 1200 nm, respectively. The second and third sets of HR and LR EBSD data are additively manufactured CoCrFeMnNi HEAs containing 1 at% carbon used in²⁶. The HR and LR EBSD were processed to have square grids with step sizes of 750 and 3000 nm, respectively. The results are represented in Fig. 9. The SRResNet super-resolved images strongly resemble HR images. In contrast, grain boundaries of microstructure images upsampled through conventional techniques are either jagged or jagged and blurry. The SRResNet not only successfully super-resolves experimentally acquired LR EBSD microstructure images, but also poses a clear advantage over conventional techniques. Furthermore, the success of super-resolving LR images using the SRResNet heavily relies on the data used to train the network. The synthetic microstructures used for training the SRResNet were noise-free and single orientation per grain. Interestingly, one can see from the triangle region in Fig. 9, in-grain misorientation is super-resolved as long as the LR EBSD data can resolve some level of in-grain misorientation. Also, as shown in the circled region in Fig. 9, the SRResNet might treat the noise as very small grains. Thus, denoising filters or pre-processing of the EBSD data is required when a significant level of noise is present. The SRResNet, however, cannot super-resolve information that cannot be resolved from LR EBSD data. One can see from the squared region in Fig. 9, LR EBSD data failed to detect small twin boundaries. This limits the performance of the SRResNet as shown in the squared region in Fig. 9.

Fig. 9: Experimentally obtained high-resolution, experimentally obtained low-resolution, and super-resolved inverse pole figure (IPF) maps.

From left to right are of CoCrFeMnNi²⁴ and x-plane and z-plane of the additively manufactured CoCrFeMnNi with 1 at% C²⁶. The x-direction is the laser scan direction and the z-plane is the building plane for the additively manufactured CoCrFeMnNi with 1 at% C.

The SRResNet is not trained to handle information related to confidence index (CI) image quality (IQ), or band contrast that can be used to identify martensite and bainite structure in steels. Because the SRResNet discerns one phase from another via phase map, phase segmentation based on CI values³³ by processing raw EBSD data will be necessary prior to super-resolving steel microstructures with martensite or bainite structures.

Methods

Collection of synthetic microstructure data

In general, the development of a deep neural network (DNN) is hampered by the limited size of the experimental data available. Recent efforts to overcome the limitation have been based on producing a synthetic database that strongly resembles experiment ones^34,35. In particular, Cecen et al.³⁴. successfully trained DNNs using a synthetic microstructure database. In this work, a database consisting of dual-phase microstructures to train SRResNet was created using the open-source software DREAM.3D³⁶. Data acquired using EBSD typically consists of phase and orientation data. Within an EBSD data, microstructural features such as grains, crystallites, phases, inclusions, and interfaces are distinguished from one another by Euler angles and phase indices. There are two methods of attaining EBSD microstructure data. The first is to access IPF or grain-based RGB colored-coded images with phase maps through published literature such as Fig. 1a. The second approach is to obtain actual or processed EBSD data with orientation information. We have trained our SR network with IPF images as well as images with RGB values obtained by multiplying Euler angles by 255/360 so that the network can super-resolve LR EBSD data acquired from either of the methods. Examples of the resulting images with Euler angle and phase information are shown in Fig. 1a.

In order to train the SRResNet, a database of microstructure images with various features is necessary. In this work, a database of two-phase microstructure images composed of 1000 × 1000 pixels was generated using open-source software DREAM.3D³⁶. The volume fraction, grain size distribution, shape, aspect ratio, and orientation of each phase were adjusted to create diverse classes of microstructures.

Two sets of microstructures were generated. The first set was composed of two primary phases, and the other set consisted of one primary phase and one precipitation phase. The primary phase was produced with a mean value of grain size in the range of 20–200, with standard deviations randomly set between 10% and 50% of the determined grain size. The minimum ($\sigma _{{\mathrm{min}}}$) and maximum ($\sigma _{{\mathrm{max}}}$) cut-off values in size distribution were fixed to 3.0 for both. The aspect ratio of a 3D particle is defined as $R_1:R_2:R_3(R_1 \ge R_2 \ge R_3)$ where $R_i$ is the length of a particle along i-axis. To adjust the aspect ratio of the primary phase, $R_1$ was decided to be a random value between 1 and 30, while R₂ and R₃ were fixed to one. The phase shape was randomly selected from three sets: ellipse, super-ellipse, and cube octahedron. Since the 2D image would be extracted in the z-plane direction, particle orientation was controlled by changing only φ, or the rotation around the z-axis in Euler angles. The rotation angle φ was randomly set in the range of 0° to 180°, and the weight of the orientation was adjusted between 1 and 10⁵. In the case of two primary phases, the volume fraction of one phase was randomly set between 0 and 50%, while the other phase occupied the entire remaining volume. In the case of precipitation, all conditions of microstructural features were the same as the conditions for the generation of the primary phase, except for mean grain size and phase fraction. Precipitation is generated by overlaying on the already created primary phase, so it is generally appropriate for a phase of smaller size and volume. Therefore, the mean value of grain size of precipitation was set to be in the range of 10–100 μm, which is half of the primary phase, and the volume fraction was set between 0 and 25%. It should be noted that the synthetic microstructures assumed a noise-free single orientation per grain state. Thus, in-grain misorientation and noisy scan points due to low CI were not taken into account in training the SRResNet.

Data preprocessing

DNNs require a large amount of data for training, in which the input data are IPF or orientation and phase images in LR, and the target data are corresponding images in HR. Figure 1a, b shows HR microstructure images produced by DREAM.3D, and the process of generating LR images using nearest neighbor downsampling, respectively. It should be noted that we do not use an anti-aliasing filter in the downsampling process of EBSD images at the expense of distortion artifacts since an anti-aliasing filter transforms jagged edges into blurred edges that are not an inherent characteristic of EBSD images. We have collected 12,000 pairs of LR and HR images for each of three downsampling factors, 4, 8, and 16. The pairs are separated by 80%, 10%, and 10% for training, validation, and test, respectively. We preprocess the pairs before training our network. First, the pixel values of all the pairs are scaled in the range of [0,1] to stabilize the learning process. Then, we randomly crop the LR images and the corresponding HR images, which are sized $224/r \times 224/r$ and $224 \times 224$, respectively, where r is the downsampling factor. In the training stage, the cropped low and HR patches are used as input and target data, respectively. Note that we directly feed any size of LR images into our trained network during testing.

Image similarity measurement

To quantitatively test image similarity measures, we used two well-known IQ metrics: the peak-signal-to-noise ratio (PSNR) and the SSIM. PSNR shows the ratio of the signal’s maximum power (denoted as MAX) to its noise (denoted as MSE). MAX is equal to 255 when image pixels are represented by an 8-bit resolution. MSE is defined as

$${\mathrm{MSE}} = \frac{1}{{mn}}\mathop {\sum}\limits_{i = 0}^{m - 1} {\mathop {\sum}\limits_{j = 0}^{n - 1} {\left[ {x\left( {i,j} \right) - y\left( {i,j} \right)} \right]^2,} }$$

(2)

where x and y represent original and degraded image data, respectively, and m and n represent the number of rows and columns, respectively.

PSNR is expressed as

$${\mathrm{PSNR}} = 20 \cdot \log _{10}\left( {\frac{{{\mathrm{MAX}}}}{{\sqrt {{\mathrm{MSE}}} }}} \right)$$

(3)

SSIM is composed of three comparative measurements: luminance, contrast, and structural, as follows:

$$l\left( {{\mathrm{x}},{\mathrm{y}}} \right) = \frac{{2\mu _x\mu _y + c_1}}{{\mu _x^2 + \mu _y^2 + c_1}},$$

(4)

$$c\left( {{\mathrm{x}},{\mathrm{y}}} \right) = \frac{{2\sigma _x\sigma _y + c_2}}{{\sigma _x^2 + \sigma _y^2 + c_2}},$$

(5)

$$s\left( {{\mathrm{x}},{\mathrm{y}}} \right) = \frac{{\sigma _{xy} + c_3}}{{\sigma _x\sigma _y + c_3}},$$

(6)

where $\mu _x$ and $\mu _y$ denote the average of x and y, $\sigma _x$ and $\sigma _y$ denote the variance of x and y, respectively, and $\sigma _{xy}$ denote covariance of x and y. The constant $c_1$, $c_2$, $c_3$ are used to stabilize the division. SSIM is then a weighted combination of the three comparative measurements, as follows:

$${\mathrm{SSIM}}\left( {x,y} \right) = \left[ {l\left( {{\mathrm{x}},{\mathrm{y}}} \right) \cdot {\mathrm{c}}\left( {{\mathrm{x}},{\mathrm{y}}} \right) \cdot {\mathrm{s}}\left( {{\mathrm{x}},{\mathrm{y}}} \right)} \right]$$

(7)

Micromechanical modeling

A crystal plasticity model is used to assess the micromechanical behavior of LR, HR, and super-resolved microstructures of Al_0.5CoCrFeMnNi HEA²³. The LR microstructure is generated by the nearest neighbor downsampling the HR microstructure. The crystal plasticity framework used in this work is based on the work by Kalidindi et al.³⁷. In the framework, slip is incorporated as

$${\mathbf{L}}_{\mathrm{p}} = \mathop {\sum}\limits_\alpha ^{N^{\mathrm{s}}} {\dot \gamma ^\alpha {\mathbf{S}}_{{\mathrm{slip}}}^\alpha } ,$$

(8)

$${\mathbf{S}}_{{\mathrm{slip}}}^{\upalpha} = {\mathbf{M}} \otimes {\mathbf{N}},$$

(9)

where, ${\mathbf{L}}_{\mathrm{p}}$, $\dot \gamma ^\alpha$, $N^{\mathrm{s}}$, and ${\mathbf{S}}_{{\mathrm{slip}}}^\alpha$ denote the plastic velocity gradient, the plastic shearing rate in slip system α, the total number of slip systems, and Schmid tensor with slip direction M and slip plane N, respectively. The plastic shearing rate in slip system α can be described using a power-law form³⁷:

$$\dot \gamma ^\alpha = \dot \gamma _0\left| {\tau ^\alpha /s^\alpha } \right|^{1/m}{\mathrm{sign}}\left( {\tau ^\alpha /s^\alpha } \right),$$

(10)

here, $\dot \gamma _0$, $\tau ^\alpha$, $s^\alpha$, and m refer to the reference shear rate, resolved shear stress, slip resistance of the α slip system, and strain rate sensitivity, respectively. The hardening law associated with each slip system can be described as

$$\dot s^\alpha = h_{\mathrm{s}}(1 - s^\alpha /s_{{\mathrm{s}}0})\mathop {\sum }\limits_{{\mathrm{slip}}} \dot \gamma ^k$$

(11)

where $\dot s^\alpha$, $s_{{\mathrm{s}}0}$, and $h_{\mathrm{s}}$ refer to the change in the slip resistance rate, the saturation value associated with slip system $\alpha$, and the hardening rate, respectively.

The FE simulation was conducted using the commercial ABAQUS (ver. 6.9. EF2) standard software. The EBSD data and hardening curve used for the crystal plasticity constitutive equations were based on those for HEA²³. The EBSD data was processed to remove in-grain misorientation. The parameters for the crystal plasticity constitutive equation are shown in Table 3. The elastic constants of the primary FCC phase in Table 3 were obtained from ref. ³⁸. The second phase was assumed to undergo isotropic J2 plasticity without hardening. The constitutive parameters of the second phase included elastic modulus, Poisson ratio, and yield strength of 210 GPa, 0.3, and 2141 MPa²⁵, respectively. A two-dimensional plane strain condition with displacement periodic boundary conditions was imposed for all simulations.

Table 3 Parameters for crystal plasticity model.