In recent years many significant advances have been made in developing numerical optimization algorithms for large-scale machine learning applications, typically deep learning. Momentum techniques (MT) are widely imposed into various optimization approaches due to its efficiency of increasing convergence speed, dampening oscillations, and avoiding local minima or saddle points. However, because of the complexity, time and expense involved in training a deep neural network, research on using MT stays on the framework of the stochastic gradient descent (SGD) algorithm. In this work, we introduce MT into the traditional non-linear conjugate gradient and quasi-Newton optimization methods, which combines the advantages of both MT and traditional optimization methods. Meanwhile, we propose a descent direction memory (DDM) method based on the essential idea of MT. We validate the use of MT and the proposed DDM method using a classical performance test problem and a 1D seismic inversion example. The experiments show off the combined effects of MT, DDM, and traditional optimization methods in generally increasing convergence rate and obtaining a smaller steady-state error.