Effects of electron correlations on a two dimensional quantum spin Hall (QSH)
system are studied. We examine possible phases of a generalized Hubbard model
on a bilayer honeycomb lattice with a spin-orbit coupling and short range
electron-electron repulsions at half filling, based on the slave rotor
mean-field theory. Besides the conventional QSH phase and a broken-symmetry
insulating phase, we find a new phase, a fractionalized quantum spin Hall
phase, where the QSH effect arises for fractionalized spinons which carry only
spin but not charge. Experimental manifestations of the exotic phase and
effects of fluctuations beyond the saddle point approximation are also
discussed.