Exposure, or adaptation, to faces or voices biases perceptions of subsequent stimuli, for example, causing faces to appear more normal than they would be otherwise if they are similar to the previously presented stimuli. Studies also suggest that there may be cross-modal adaptation between sound and vision, although the evidence is inconsistent. We examined adaptation effects within and across voices and faces and also tested whether adaptation crosses between male and female stimuli. We exposed participants to sex-typical or sex-atypical stimuli and measured the perceived normality of subsequent stimuli. Exposure to female faces or voices altered perceptions of subsequent female stimuli, and these adaptation effects crossed modality; exposure to voices influenced judgments of faces, and vice versa. We also found that exposure to female stimuli did not influence perception of subsequent male stimuli. Our data demonstrate that recent experience of faces and voices changes subsequent perception and that mental representations of faces and voices may not be modality dependent. Both unimodal and cross-modal adaptation effects appear to be relatively sex-specific.