The "uncertainties" in contemporary scientific data are not as simplistic as additive Gaussian noise, particularly with datasets arising from geophysical settings. Instead, the noise interacts with the forward model in a far more coherent, nonlinear manner. To that end, we propose capturing these uncertainties with "nuisance parameters" embedded in the physical model. This results in challenging inverse problems for which variational auto-encoders (VAEs) and unsupervised learning are well-suited for solving.