Exclusion by outgroups is often attributed to external factors such as prejudice. Recently, event-related potential studies have demonstrated that subtle cues influence expectations of exclusion, altering the P3b response to inclusion or exclusion. We investigated whether a visual difference between participants and interaction partners could activate expectations of exclusion, indexed by P3b activity, and whether this difference would influence psychological responses to inclusion and exclusion. Participants played a ball-tossing game with two computer-controlled coplayers who were believed to be real. One period involved fair play inclusion while the other involved partial exclusion. Avatars represented participants, with their color matching participant skin tone, and either matching or differing from the color of coplayer avatars. This created the impression that the participant was an ingroup or outgroup member. While ingroup members elicited enhanced P3b activation when receiving the ball during exclusion, outgroup members showed this pattern for both inclusion and exclusion, suggesting that they formed robust a-priori expectations of exclusion. Self-reports indicated that while these expectations were psychologically protective during exclusion, they were detrimental during inclusion. Ultimately, this study reveals that expectations of exclusion can be formed purely based on visual group differences, regardless of the actual minority or majority status of individuals.