A framework for designing mimo systems with decision feedback equalization or tomlinson-harashima precoding

We consider joint transceiver design for point-to-point Multiple-Input Multiple-Output communication systems that implement interference (pre-)subtraction; i.e., Decision Feedback Equalization (DFE) or Tomlinson-Harashima precoding (THP). We develop a unified framework for joint transceiver design of these two dual systems by considering design criteria that are expressed as functions of the (logarithm of the) Mean Square Error (MSE) of the individual data streams. By deriving two inequalities that involve the logarithms of the individual MSEs, we obtain optimal designs for two broad classes of communication objectives, namely those that are Schur-convex and Schur-concave functions of these logarithms. These two classes embrace several design criteria for which the optimal transceiver design has remained an open problem. For Schur-convex objectives, the optimal design results in data streams with equal MSEs. In addition to other desirable properties, this design simultaneously minimizes the total MSE and the average bit error rate, and maximizes the Gaussian mutual information; a property that is not achieved by a linear transceiver. Moreover, we show that the optimal design yields objective values that are superior to the corresponding optimal objective value for a linear transceiver. For Schur-concave objectives, the optimal DFE design results in linear equalization and the optimal THP design results in linear precoding. The proposed design framework can be regarded as a counterpart of the existing framework for linear transceiver design.