Mou and McNamara have suggested that object locations are represented according to intrinsic reference frames. In three experiments, we investigated the limitations of intrinsic reference frames as a mean to represent object locations in spatial memory. Participants learned the locations of seven or eight common objects in a rectangular room and then made judgments of relative direction based on their memory of the layout. The results of all experiments showed that when all objects were positioned regularly, judgments of relative direction were faster or more accurate for novel headings that were aligned with the primary intrinsic structure than for other novel headings; however, when one irregularly positioned object was added to the layout, this advantage was eliminated. The experiments further indicated that with a single view at study, participants could represent the layout from either an egocentric orientation or a different orientation, according to experimental instructions. Together, these results suggest that environmental reference frames and intrinsic axes can influence performance for novel headings, but their role in spatial memory depends on egocentric experience, layout regularity, and instructions.