Léger et al. [J. Acoust. Soc. Am. (2012)] measured the intelligibility of speech in steady and spectrally or temporally modulated maskers for stimuli filtered into low- (<1.5 kHz) and mid-frequency (1–3 kHz) regions. Listeners with high-frequency hearing loss but near to clinically normal audiograms in the low- and mid-frequency regions showed poorer performance than a control group with normal hearing, but showed preserved spectral and temporal masking release. Here, we investigated whether a physiologically accurate model of the auditory periphery [Zilany et al., J. Acoust. Soc. Am. (2009)] can explain these masking release data. Intelligibility was predicted using the Neurogram SIMilarity (NSIM) metric of Hines and Harte [Speech Commun. (2010) and (2012)]. This metric can make use of either an “all-information” neurogram with small time bins or a “mean-rate” neurogram with large time bins. The average audiograms of the different groups of listeners from the study of Léger et al. were simulated in the model by applying different mixes of outer and/or inner hair cell impairment. Very accurate predictions of the human data for both normal-hearing and hearing-impaired groups were obtained from the all-information NSIM metric (i.e., taking into account phase-locking information) with threshold shifts produced predominantly by OHC impairment (and minimal IHC impairment).