This paper outlines the challenges faced during direct built environment (BE) assessments of 42 Canadian communities of various income and urbanization levels. In addition, we recommend options for overcoming such challenges during BE community assessments. Direct BE assessments were performed utilizing two distinct audit methods: (1) modified version of Irvine-Minnesota Inventory in which a paper version of an audit tool was used to assess BE features and (2) a Physical Activity and Nutrition Features audit tool, where the presence and positions of all environmental features of interest were recorded using a Global-Positioning-System (GPS) unit. This paper responds to the call for the need of creators and users of environmental audit tools to share experiences regarding the usability of tools for BE assessments. The outlined BE assessment challenges plus recommendations for overcoming them can help improve and refine the existing audit tools and aid researchers in future assessments of the BE.