|Currently, the 3rd Generation Partnership Project (3GPP) standards body is working actively to specify Voice services over Adaptive Multi-user channels on One Slot (VAMOS), an enhanced transmission scheme to double the voice service capacity in existing Global System for Mobile communications (GSM) system. Capacity increase is achieved by multiplexing two users in the same timeslot within the same radio frequency channel. With this new transmission scheme, several characteristics of VAMOS downlink signals can be taken into account in mobile station equalizer design. Firstly, the base station maps a pair of bits from the two VAMOS users to a quaternary symbol whose I/Q amplitude ratio may vary burst by burst. When the ratio is not unity, the VAMOS downlink signal is improper because its pseudo-covariance does not vanish. Secondly, in the baseband modulation stage, the symbol sequence is processed by π/2 progressive symbol rotation and linearized Gaussian minimum shift keying (LGMSK) pulse shaping. Hence, intersymbol interference (ISI) is introduced. Lastly, due to tight frequency reuse in most GSM networks today, mobile stations also experience co-channel interference (CCI) from other base stations transmitting various types of signals such as GMSK, 8 phase shift keying (8PSK), or VAMOS. In order to achieve network capacity increase, advanced equalizers in mobile stations are required to offer superior bit error rate (BER) performance in this challenging mobile radio system.
Unlike uplink transmission where a base station is typically equipped with an antenna array to enhance the desired signal and suppress CCI, equalizer design in downlink direction is more challenging because most mobile stations have one antenna only. In the past decade, various single antenna interference cancellation (SAIC) solutions have been developed to mitigate the CCI effect. While these solutions can be straightforwardly reused to obtain acceptable performance in VAMOS, in this work a novel equalizer is proposed to achieve better BER performance by addressing the forementioned characteristics of VAMOS downlink signals. This equalizer consists of a widely linear multiple-input multiple-output (MIMO) minimum mean square error (MMSE) filter and a joint maximum likelihood sequence estimator (JMLSE). In the first stage, since the transmitted signal and the received signal may not be jointly proper, a widely linear filter is more effective in suppressing interference from other base stations plus background noise while preserving the two paired VAMOS users’ signal energy. In the next stage, further interference cancellation is achieved by jointly detecting the two users’ symbol sequences using JMLSE. Optimality of the proposed equalizer is justified by theoretical proof and simulation in this work.