A complex picture of factors influencing cognition is necessary to be drawn for a better understanding of the role of potentially modifiable factors in dementia. The aim was to assess the prevalence and determinants of cognitive impairment, including the role of cerebral small vessel disease (CSVD) in Polish middle-aged cohort. A comprehensive set of clinical (hypertension, coronary heart disease, diabetes mellitus, hyperlipidaemia, body mass index, smoking status, alcohol intake) and socio-demographic data was collected in the PURE study in years 2007–2016, which was the basis for detailed analysis of risk factors of cognitive impairments in years 2016–2018 in the PURE-MIND sub-study. Five hundred forty-seven subjects (age range 39–65, mean 56.2 ± 6.5) underwent neuropsychological assessment with Montreal Cognitive Assessment (MoCA), Trail Making Test (TMT) and Digit Symbol Substitution Test (DSST) followed by brain MRI. Mean MoCA score was 26.29 and 33% participants met criteria for mild cognitive impairment (MCI) (MoCA< 26). Seventy-three percent showed findings related to CSVD. Higher WMH burden and lacunar infarcts were associated with lower MoCA and DSST scores. Severe CSVD was associated with twofold incidence of MCI, and obesity increased its probability by 53% and hypertension by 37%. The likelihood of MCI was reduced in nonsmokers. One factor analysis showed the important role of lower level of education, older age, rural area of residence and hypertension. MCI and CSVD are highly prevalent in the middle-aged population in Poland. A greater importance should be given to potentially modifiable risk factors of dementia which are already present in mid-life.