There is a great individual difference in people’s face recognition ability (FRA). This study aimed to reveal the neural mechanism underlying such individual differences. Elastic-net regression models were constructed to predict FRA based on the white matter (WM) microstructural properties. We found that FRA can be accurately predicted by the WM microstructural properties. For the right inferior longitudinal fasciculus (ILF) and bilateral arcuate fasciculus (AF), FRA was correlated negatively to fractional anisotropy (FA), but positively to radial diffusivity (RD). In contrast, for the corpus callosum forceps minor (CFM), FRA was correlated positively to FA, but negatively to RD. Such various patterns of the WM microstructural properties suggested a positive correlation between FRA and fiber diameter for the right ILF and bilateral AF, but a negative correlation between FRA and diameter of the CFM. These findings reflected that FRA was correlated positively to connectivities of the right ILF and bilateral AF, but negatively to those of the CFM. These findings not only confirmed the significant role of the right ILF in face recognition, but also revealed the involvement of the bilateral AF and CFM in face recognition, particularly implying the important role of hemisphere lateralization modulated by transcallosal connectivity in face recognition.