In this paper, the transmission of an improper-complex second-order stationary data sequence is considered over a strictly band-limited frequency-selective channel. It is assumed that the transmitter employs linear modulation and that the channel output is corrupted by additive proper-complex cyclostationary noise. Under the average transmit power constraint, the problem of minimizing the mean-squared error at the output of a widely linear receiver is formulated in the time domain to find the optimal transmit and receive waveforms.
The optimization problem is converted into a frequency-domain problem by using the vectorized Fourier transform technique and put into the form of a double minimization. First, the widely linear receiver is optimized that requires, unlike the linear receiver design with only one waveform, the design of two receive waveforms. Then, the optimal transmit waveform for the linear modulator is derived by introducing the notion of the impropriety frequency function of a discrete-time random process and by performing a line search combined with an iterative algorithm. The optimal solution shows that both the periodic spectral correlation due to the cyclostationarity and the symmetric spectral correlation about the origin due to the impropriety are well exploited.