The natural frequencies and seismic-wave velocities of buildings and other civil structure are changing continuously due to earthquakes, weather especially temperature and precipitation, and aging effects. Strong motion related to earthquakes modify the seismic response of buildings. When earthquake waves are not strong (small strain), the natural frequencies nonlinearly decrease at the time of earthquakes because of shaking and bending, but it recovers quickly afterward. When earthquakes are large, strong motion gives permanent changes as a damage to the structure. For example, the 2011 M9.0 Tohoku earthquake in Japan caused about 20% of reduction in the fundamental frequency of a building, which is several hundred kilometers far from the epicenter. Even when we do not have strong earthquakes, buildings last for decades, and the natural frequencies are also changing over the time (i.e., aging). We study these effects of strong motion and aging using a building in Tsukuba, Japan. We apply spectral decomposition and deconvolution techniques to more than 1600 earthquake waveforms observed in the building over 20 years, including the 2011 Tohoku earthquake. The natural frequency and seismic velocities in the building have been decreased over the 20 years, caused by both strong earthquakes and aging. Interesting, nearby borehole seismometers do not show such large changes. The change caused by the Tohoku event is recovering slowly within two years from the mainshock; however, the response is still significantly different from the one before the earthquake. With this long-term observation, we analyze the effects of earthquakes and aging, and separate them.