This paper explores the hypothesis that the returns of asset classes can be predicted using common, systematic risk factors represented by the level, slope, and curvature of the US interest rate term structure. These are extracted using the Nelson–Siegel model, which effectively captures the three dimensions of the yield curve. To forecast the factors, we applied autoregressive (AR) and vector autoregressive (VAR) models. Using their forecasts, we predict the returns of government and corporate bonds, equities, REITs, and commodity futures. Our predictions were compared against two benchmarks: the historical mean, and an AR(1) model based on past returns. We employed the Diebold–Mariano test and the Model Confidence Set procedure to assess the comparative forecast accuracy. We found that Nelson–Siegel factors had significant predictive power for one-month-ahead returns of bonds, equities, and REITs, but not for commodity futures. However, for 6-month and 12-month-ahead forecasts, neither the AR(1) nor VAR(1) models based on Nelson–Siegel factors outperformed the benchmarks. These results suggest that the Nelson–Siegel factors affect the aggregate stochastic discount factor for pricing all assets traded in the US economy.