People with visible differences are often confronted with negative observer responses, including stares, disgust, and avoidance. Characteristics of negative observer responses are well-documented, but less is known about associated factors. We conducted a scoping review to map what is known about factors associated with negative observer responses. Web of Science, PubMed, PsycINFO, and CINAHL databases were searched, and 16 articles met inclusion criteria. Two general categories of factors were identified: (1) observer characteristics, including age, sex, and socioeconomic status, experiences with disfigurements, and personal beliefs or attitudes related to visible differences; and (2) evolved internal mechanisms, including threat-detection, disgust, and disease avoidance. Additionally, there was evidence that lack of anonymity influences lower reporting of observer reactions. Efforts that increase exposure to individuals with visible differences may ameliorate adverse reactions; however, due to the limited nature of evidence reviewed, further research is needed before more concrete recommendations can be made.