Predator density and competition modify the benefits of group formation in a shoaling reef fish

Synthesis Predation risk experienced by individuals living in groups depends on the balance between predator dilution, competition for refuges, and predator interference or synergy. These interactions operate between prey species as well: the benefits of group living decline in the presence of an alternative prey species. We apply a novel model‐fitting approach to data from field experiments to distinguish among competing hypotheses about shifts in predator foraging behavior across a range of predator and prey densities. Our study provides novel analytical tools for analyzing predator foraging behavior and offers insight into the processes driving the dynamics of coral reef fish. Studies of predator foraging behavior typically focus on single prey species and fixed predator densities, ignoring the potential importance of complexities such as predator dilution; predator‐mediated effects of alternative prey; heterospecific competition; or predator–predator interactions. Neglecting the effects of prey density is particularly problematic for prey species that live in mixed species groups, where the beneficial effects of predator dilution may swamp the negative effects of heterospecific competition. Here we use field experiments to investigate how the mortality rates of a shoaling coral reef fish (a wrasse: Thalassoma amblycephalum ), change as a result of variation in: 1) conspecific density, 2) density of a predator (a hawkfish: Paracirrhites arcatus ), and 3) presence of an alternative prey species that competes for space (a damselfish: Pomacentrus pavo ). We quantify changes in prey mortality rates from the predator's perspective, examining the effects of added predators or a second prey species on the predator's functional response. Our analysis highlights a model‐fitting approach that discriminates amongst multiple hypotheses about predator foraging in a community context. Wrasse mortality decreased with increasing conspecific density (i.e. mortality was inversely density‐dependent). The addition of a second predator doubled prey mortality rates, without significantly changing attack rate or handling time – i.e. there was no evidence for predator interference. The presence of a second prey species increased wrasse mortality by 95%; we attribute this increase either to short‐term apparent competition (predator aggregation) or to a decrease in handling time of the predator (e.g. through decreased wrasse vigilance). In this system, 1) prey benefit from intraspecific group living though a reduced predation risk, and 2) the benefit of group living is reduced in the presence of an alternative prey species.