Estimating Q based on downgoing waves in VSP data can be challenging when scattered waves from near-borehole velocity/density anomalies interfere constructively and destructively with the direct arrivals. The wavefield interference leads to modulations in amplitude spectra and these undesired spectral changes result in over- and/or under-estimates of Q values. In order to improve the reliability and accuracy of Q estimates, we propose locating such scatterers and mitigating their effects from the spectra by combining waveform modelling and adaptive subtraction. We investigate this wavefield interference phenomenon and test our proposed method using a zero offset VSP field dataset from offshore Australia and synthetic data created based on well logs in the area.