Early detection of dementia provides opportunities for interventions that could delay or prevent its progression. We developed the Japanese version of the Quick Mild Cognitive Impairment (Qmci-J) screen, which is a performance-based, easy-to-use, valid and reliable short cognitive screening instrument, and then we examined its validity. Community-dwelling adults aged 65–84 in Niigata prefecture, Japan, were concurrently administered the Qmci-J and the Japanese version of the standardized Mini-Mental State Examination (sMMSE-J). Mild cognitive impairment (MCI) and dementia were categorized using established and age-adjusted sMMSE-J cut-offs. The sample (n = 526) included 52 (9.9%) participants with suspected dementia, 123 (23.4%) with suspected MCI and 351 with likely normal cognition. The Qmci-J showed moderate positive correlation with the sMMSE-J (r = 0.49, p < 0.001) and moderate discrimination for predicting suspected cognitive impairment (MCI/dementia) based on sMMSE-J cut-offs, area under curve: 0.74, (95%CI: 0.70–0.79), improving to 0.76 (95%CI: 0.72 to 0.81) after adjusting for age. At a cut-off of 60/61/100, the Qmci-J had a 73% sensitivity, 68% specificity, 53% positive predictive value, and 83% negative predictive value for cognitive impairment. Normative data are presented, excluding those with any sMMSE-J < 27. Though further research is required, the Qmci-J screen may be a useful screening tool to identify older adults at risk of cognitive impairment.