Spatial frames of reference in human attention have been well-studied for processing visual events in two-dimensional (2D) space, but not in three-dimensional (3D). Through the inhibition of return (IOR) effect in a modified spatial cueing paradigm, we have previously demonstrated a near-advantage in localization performance, and we identified that viewer-centered distance modulated spatial attention. In the current study, we examined the separate contributions of viewer-centered distance and world-centered depth on the IOR effect. We compared conditions where viewer-centered distance and world-centered depth could specify the same or different spatial relation between the cue and target. The results showed that IOR decreased in conditions with different cue/target world-centered depths but the same viewer-centered distance. However, IOR remained large when the cue/target stimuli appeared within the same world-centered depth, regardless of whether the cue/target held the same or different viewer-centered distance and was either within (Experiment 1) or beyond (Experiment 2) the same placeholder. The results suggested that when the target appears in a different world-centered depth from that of the cue, both viewer-centered distance and world-centered depth modulate spatial attention. However, when the target appears in the cued world-centered depth, inhibition is tagged to that depth plane in a winner-take-all manner.