It’s one thing investment advisors and their clients have in common. They both wonder about the future direction of the stock market. Often the focus is on the short term. What’s the market going to do next month? Next quarter? Next year? These debates are fun, but they amount to little more than a guessing game. The market is full of too many surprises.

Discussions about the long-term direction of the stock market may not be as much fun, but they are more important. High return expectations for stocks mean clients can plan to retire earlier or withdraw more from their investment portfolios each month. Low return expectations mean that clients must plan to work a little longer or draw less from their portfolios for living expenses each month. You cannot provide solid guidance to your clients without reasonable expectations about the likely future returns of stocks.

The world is full of opinions about the returns you should expect from the U.S. stock market. Most of them are derived from the study of historic returns. In his book Stocks for the Long Run, Jeremy Siegel tells us that the long-term real rate of return (after inflation) for U.S. stocks has been 7%. His opinion is based on a study of stock market returns going back to the days of the Louisiana Purchase. With the weight of this historical record, Siegel asserts that the return of the stock market going forward should also be 7%. There are those who disagree with Siegel’s assessment, yet many people still think the range is 6% to 7% annualized.

But clearly, the stock market does not provide a consistent return every year. If you expect to get Siegel’s 7% year-in and year-out, or any other static return number for that matter, you will be very disappointed. Sometimes the return of the stock market is higher and sometimes it is lower, but it is rarely, if ever, equal to its long-term historical average.

Our research suggests, instead, that the future long-term real rate of return of the U.S. stock market is actually closer to 5%. This conclusion does not flow from a more granular slicing and dicing of historical data. Rather, it flows from our effort to identify a reliable method for finding future expected returns when markets are always changing. Our goal was to find a better way than to simply use historical averages.

The starting place for our quest was the Gordon growth model, a classic model for fundamental stock valuation. Our examination of that model led us to conclude that the earnings yield (the inverse of the more popular price-to-earnings ratio) might provide a good estimator of future stock market returns.

Others have also advanced the earnings yield as an indicator. In fact, Siegel mentions in Stocks for the Long Run that earnings yield is a valid approximation of the stock market’s future expected real returns.

So our belief in the importance of earnings yield is certainly not novel. However, our efforts to confirm its usefulness as a predictor led us to some unique insights about the long-term expected return of the U.S. stock market in general and to our view that, currently, 5% is a good estimate for the real rate.
To reach our conclusions, first we calculated the P/E and earnings yield for the S&P 500 index for every month from the beginning of 1926 through the end of 2012. We used the index value to represent the price of the market. That information is widely available and easy to acquire. Determining the earnings of the market was a little more complicated.

It is common for stock market analysts to track the last 12 months of earnings for the market and make short-term market forecasts based on these reports. The problem is that short-term earnings reports are subject to future accounting adjustments, and earnings can vary significantly year to year with world events. In his book Irrational Exuberance, Robert Shiller suggests that normalizing earnings using a 10-year inflation-adjusted average (“P/E 10”) helps smooth out these accounting adjustments and the effects of short-term boom/bust cycles. We used Shiller’s P/E 10 approach to derive our historical earnings yield for the S&P 500 index.

Then we looked for time periods that would allow us to test the predictive value of the 10-year normalized earnings yield as an indicator. In order to avoid problems with end-point bias, we looked for time periods where the beginning, ending and average P/E 10 were all the same. Peter Bernstein suggested a similar approach in his article, “What Rate of Return Can You Reasonably Expect … Or What Can the Long Run Tell Us About the Short Run?” (which appeared in the March/April 1997 issue of the Financial Analysts Journal). As it turns out, there have been only two such periods since 1926.
The first period was March 1927 through July 1954. The P/E 10 at the beginning of the period was 14. It was 14.1 at the end of the period, and the average annual P/E 10 during this time was also 14. That translates into an earnings yield of 7.1%.

If earnings yield is a good predictor of future stock market returns, we would expect to see annualized real returns of about 7.1% over the relevant time period. The actual return during that period was 7.1%.

It is interesting to note that this finding is consistent with Jeremy Siegel’s 7% estimate for the real rate of return for U.S. stocks.

The second time period was March 1961 through January 2010. The P/E 10 at the beginning and end of the period was 19.6, while the average annual P/E 10 through these years was 19.4. This translates into an earnings yield of 5.1%, so if this is a good predictor of future real returns, we would expect to see annualized real returns of about 5.1%. The actual return during that period was 5.0%—very close.

Our findings prompted us to ask: Why did the earlier time period line up so well with Siegel’s 7% prediction, but the latter period, almost 50 years in length, fall so far short? Had something changed?

The annualized real rate of return for the S&P 500 during the entire period from 1926 through 2012 was 6.7%, close to Siegel’s 7%. But when we looked closer, we found that this 87-year period actually encompasses three distinct, shorter periods where the annualized real rate of return was different. The rate was approximately 7% from 1927 to 1954, 14% from 1955 to 1960 and 5% from 1961 to 2010.

We believe people’s expectations about real return for the U.S. stock market shifted during these periods. We are not entirely sure why, but we surmise that investors simply came to see stocks as less risky than they had been, and ultimately this perception was correct. The standard deviation of U.S. stocks from 1926 through 1954 was 29.6%, but it was only 16.3% from 1961 through 2012. Perhaps the advent of modern portfolio theory caused people to realize that the volatility of a single stock could be offset by combining it more scientifically with a basket of other stocks. Maybe the increase in U.S. government spending following World War II and the emergence of the U.S. as a major economic power assured people that the economy would be more stable in the future.

Whatever the reasons, beginning in 1961 (after an apparent six-year transition period), stocks were priced as though they were less risky. Because they were.

We believe that Siegel and others may have overlooked this shift in the future expected return of U.S. stocks because they focused too much on long-term historical averages. Siegel’s efforts to analyze stock market returns back to 1802 are impressive. But too much historical data can also mask important changes.

It is worth emphasizing that we are not advocating a better, more thorough analysis of long-term historical returns. Using these to estimate future stock market returns is simply an ill-advised practice. For example, if you look at the 50-year period from 1950 through 1999, you will find that the annualized real return of the U.S. stock market was 9.2%. If you roll the clock forward only 12 years and look at the 50-year period from 1962 through 2011, you find an annualized real return of only 4.95%–and a difference of more than 4% per year in these averages. How do you determine which one is “right” going forward?

One way is to derive your estimates using an approach based on normalized earnings yield. Using the same two time periods as examples, the 10-year normalized earnings yield at the beginning of 1950 was 9.4%—very close to the actual annualized return of 9.2% that the market generated over the next 50 years. The 10-year normalized earnings yield at the beginning of 1962 was 4.6%—very close to the actual return of 4.95%. It’s not perfect, but using an earnings yield approach for estimating stock market returns certainly beats a static historical average.

Here are a few cautionary observations about using earnings yield to forecast future stock market returns. First, the 10-year normalized earnings yield is in a constant state of flux. Even though we believe that, currently, the long-term average earnings yield is about 5%, there are frequent variations around that number. In managing portfolios, we look at the normalized earnings yield for the U.S. stock market monthly and make any necessary adjustments.

Furthermore, we have found that the duration of the forecast varies as the earnings yield varies. It appears that a current 10-year normalized earnings yield of 5% allows a forecast of about 20 years. As the earnings yield increases, the duration of the forecast is shortened. A lower earnings yield increases the duration of the forecast.

Finally, it would be unwise to think the long-term real rate of return for U.S. stocks will remain 5% forever. It appears to have already changed once during our lifetime and there is no reason to believe it won’t change again. Fundamental long-term improvements in the global economic environment could cause the future expected returns to decline in the face of perceived lower risk. On the other hand, a profound deterioration of global economic fundamentals could cause an increase in expected returns. It is always good policy to continually test your assumptions about the future.

There are significant implications for financial advisors if we are correct in asserting that, at least for now, the long-term future expected real return of U.S. stocks is 5%. If the market is trading at a P/E of 20, we would say that it is right at its long-term average. If you believe the true long-term real return of U.S. stocks is 6.25% (a P/E of 16) you would think the market is trading at 25% above its average. That’s a big difference. Also, if you are preparing a financial plan based on the assumption that your client will receive a real return of 6.25% from U.S. stocks and the market returns only 5%, your clients will fall far short of the projections you have made for them. A difference of 1% or 2% in the expected return of the stock market can have a huge impact on how assets are allocated within a portfolio or on the structure of a financial plan.

Here’s another thing that advisors and their clients have in common: They both like their long-term expectations about future stock market returns to be close to the actual results. We believe that, for now, an annualized long-term real return of 5% is a reasonable expectation. Importantly, that expectation has a logical basis—earnings yield—and is not derived through the accident of sampling average returns over arbitrarily selected historical time periods. 

Gary A. Miller, CFA, is founder and chief investment officer of Frontier Asset Management. Scott A. MacKillop is President of Frontier. Frontier manages portfolios for a select group of financial advisors and their clients. Scott can be reached at smackillop@frontierasset.com.