Wind-sensitive buildings can experience excessive vibrations that cause discomfort and interruption of the activities of the inhabitants of the buildings. To ensure the desired serviceability, codes, standards, and their commentaries have proposed perception curves that limit the expected (or mean) peak acceleration or standard deviation of the acceleration of the buildings. These perception curves are developed based on perceived vibration alone and do not consider the uncertainty in structural dynamic characteristics (i.e., natural frequency of vibration and ratio of damping) and loads such as the wind loading. Therefore, the annual probability that the perception curve is not exceeded for a building whose design is based on the perception curve is unknown. In this study, serviceability design factors are calibrated for selected targeted annual probability of perception levels by considering the uncertainty in the structural dynamic characteristics, wind characteristics, as well as in the human perception of motion. These serviceability design factors are to be used with the estimated mean peak acceleration caused by along-wind and cross-wind excitations given in the commentaries of the current National building code of Canada. The use of calibrated serviceability design factors for design checking is illustrated with a numerical example.