Subsurface characterization is critical in understanding and controlling many natural and industrial processes including groundwater movement, oil extraction, and geological carbon dioxide sequestration. While recent advances in three‐dimensional (3D) imaging of core samples have enabled digital subsurface characterization, the exorbitant computational cost associated with direct numerical simulation in 3D remains a persistent challenge. In contrast, machine learning models are much more efficient, though their use in subsurface characterization is still in its infancy. Here, we introduce a novel 3D convolutional neural network (CNN) for end‐to‐end prediction of permeability, which is a fundamental characteristic of subsurface porous media. We show that increasing the dataset size and diversity, utilizing multi‐scale feature aggregation, and optimizing the network architecture elevate the model accuracy beyond that of existing state‐of‐the‐art 3D CNN models for permeability prediction. We demonstrate that the model is generalizable, and it is capable of predicting the permeability of previously unseen samples with an excellent accuracy.