Estimating Q using downgoing waves in zero-offset vertical seismic profiles (VSPs) can be challenging when scattered waves from near-borehole heterogeneities interfere with direct arrivals. In any Q estimation method that assumes a downgoing plane wave, constructive and destructive wave-mode interference can cause errors in the estimate. For example, in the spectral-ratio method, such interference modulates the amplitude spectra introducing significant variations and even nonphysical negative Q (amplification) estimates. We have investigated this phenomenon using synthetic and field data sets from offshore Australia and developed a two-step waveform-based method to characterize scattering anomalies and improve Q estimates. Waveform information is key to deal with closely spaced band-limited seismic events. First, we solve an inverse problem to locate and characterize scatterers by minimizing the traveltime and waveform misfits. Then, using the estimated parameters, we model the scatterers’ contribution to the VSP data and remove it from the observed waveforms. The resulting spectra resemble those that would have been acquired in the absence of the scatterers and are much more suitable for the spectral-ratio method. By assuming a 1D medium and a simple scatterer shape (i.e., circular), we parameterize a scattering heterogeneity using five parameters (depth, distance, size, velocity, and density) and seek a solution using a grid search to handle the nonuniqueness of the VSP inversion. Instead, adaptive subtraction is required to fine-tune the modeled interference to better fit the observation. We successfully use this method to characterize and mitigate the strongest wave interference in the field data. The final Q estimates contain milder variations and much less nonphysical negative Q⁠. Our results demonstrate that the proposed method, readily extendible to multiple scatterer cases, can locate discrete scatterers, remove the effects of their interference, and thus significantly improve the Q estimates from VSP data.