Minimum SER Zero-Forcing Transmitter Design for MIMO Channels with Interference Pre-subtraction

We consider point-to-point multiple antenna communication systems in which multiple data streams are transmitted simultaneously. We consider systems which use Tomlinson-Harashima (TH) precoding to pre-subtract the interference among these data streams at the transmitter. In a conventional Tomlinson-Harashima precoding system, transmitter feedback and receiver feedforward processing matrices are used for interference pre-subtraction and channel spatial equalization. In addition to these matrices, we consider a transmitter precoding matrix that generalizes the permutation matrix used for ordering the precoded symbols in existing designs. This extra degree of freedom offers the potential for improved performance. In particular, under a mild signal to noise ratio (SNR) constraint, we find an optimum zero-forcing precoding matrix that minimizes the average symbol error rate (SER) of the data streams subject to a transmitter power constraint. We also show that the proposed design is optimal from an average bit error rate (BER) perspective. Simulation studies show significant improvement over conventional zero-forcing Tomlinson-Harashima precoders.