Radius selection algorithms for sphere decoding

The integer least squares problem arises from many applications such as communications, cryptography, and GPS. In this paper, we consider the sphere decoding method in communication applications. One of key issues in sphere decoding is the selection of an initial radius of the search hypersphere. We first present a deterministic radius selection algorithm using the Babai estimate. However, due to the rounding errors in floating-point computation, this method may produce a too small radius and cause sphere decoding to fail to find a solution. In this paper, we perform an error analysis and propose a modified radius selection algorithm by taking computational error into account. Our numerical experiments show that this modified method achieves high success rate without compromising performance.