Avoiding Biases in TAA
Model-Building:

by Lucie Chaumeton and Gregory Connor

The new BARRA Altis System gives investment managers the opportunity to build in-house quantitative Tactical Asset Allocation (TAA) models.¹ The comprehensive database and step-by-step regression package in Altis make it easy to test the predictive power of quantitative models linking market and macroeconomic data to the returns on various asset classes within the World Markets Model.

Experienced model-builders understand the importance of out-of-sample model-testing in the development of TAA models. Usually, much of the focus of out-of-sample testing is on portfolio optimization, but in this article we highlight the importance of out-of-sample testing of explanatory power. As we discuss, the nature of TAA models means that in-sample measures of explanatory power can be badly biased. The reason for this bias is that TAA models almost always include lagged endogenous variables in one form or another. We use simulations to quantify this bias and show how to avoid the bias with a properly formulated research procedure. Out-of-sample testing of explanatory power is easily done using the Altis Performance Calculation feature.

Uncovering a bias in in-sample R²

The standard measure of explanatory power is the R-squared (R²) statistic. The R² is the ratio of explained variance to total variance in returns. One of the most widely used statistics in empirical research in economics and finance, R² is widely used to evaluate TAA models as well. Yet in the context of most TAA models, the in-sample R² statistic can be very badly biased. The problem, as we will discuss later in the article, comes from the pervasive use of lagged endogenous variables in TAA models. We first illustrate the importance of this bias by a case study using the Altis System.

In Table 1 we estimate a typical TAA model, linking asset returns to dividend yields and lagged returns. The model has the form:

where R_t is the total return on an equity index and Yld_t is the dividend yield for the same equity index. We estimate the model in Altis using increasing samples of monthly data for three major equity indexes. Although we show the results for only three countries, this empirical analysis could be applied to a wider cross-section of countries. All in-sample periods terminate in August 1994, so as to leave two years of out-of-sample data at the end of the total data sample. For this simple model, in-sample adjusted R²'s range from 0% in Japan to 37% in the U.K. It appears that we can make enormous abnormal returns with this simple model! (Note that the R² of an optimized international TAA portfolio will be higher than the R² of any of the individual national models, due to the effect of diversification. An estimated return forecasting model with an average R² across countries of 2% to 3% provides good profit potential.)

The effect on the in-sample R² of an increasing sample size hints at the presence of a bias. In a classical framework, the in-sample (adjusted) R² should not be affected by sample size. However, we find that in each country the in-sample R² tends to go down as the sample length increases: from 37% to 4% in the U.K. and from 9% to 1% in the U.S. as the sample length goes from 24 to 60 observations. This is because as the sample size shrinks, the bias gets more prominent. This is a common feature of model biases: They often tend to be worst when the sample size is small.

The importance of the bias is shown clearly by our out-of-sample findings. We calculate an out-of-sample R² using the Altis Performance feature and maintaining the model parameters constant and equal to the in-sample coefficients. Our out-of-sample period is September 1994 to August 1996, and all but one of our out-of-sample R²s drop to zero! We also calculate out-of-sample performance, allowing the model parameters to be recalculated in rolling windows, and find likewise that R²'s collapse to zero. The in-sample model which looked to be so profitable turns out to be worthless.

Measures of explanatory power with lagged endogenous variables

The bias in R² shown in Table 1 is not a new discovery², but it is a problem which has received inadequate attention in the finance literature.

Table 1: In- and out-of-sample adjusted R²—Regression of return on lagged return and lagged dividend yield

The bias is due to the presence of lagged endogenous variables in the explanatory variables. In particular, the price of the asset appears implicitly both in the asset return (the dependent variable) and in both of the explanatory variables (dividend yield and lagged return). By definition, the dividend yield is the ratio of the dividend per share to the price of a security:

The return on a share is made up of a price appreciation and a dividend component:

Note that in a regression such as (1), both the explained variable (next period's return) and the explanatory variables (dividend yield and lagged return) include the asset price in their definition. This means that both explanatory variables include a lagged endogenous variable (price), and hence that the R² is biased. The size of this bias is difficult to derive explicitly since the lagged endogenous variable appears in a non-linear form in dividend yield.

The same problem occurs in almost all TAA models, not just those which use lagged returns and dividend yields. For example, another key TAA explanatory variable is cumulative returns. We define the cumulative returns at time T over the horizon length L as:

This type of explanatory variable also includes a lagged endogenous variable, since lagged returns appear explicitly.

Measuring the bias using simulations

It is impossible to derive an explicit formula for the size of the bias in R² except for very special cases. However, simulation techniques allow us to measure the bias quite accurately for particular models. Using a random number generator, we can replicate the logical relationship between returns, lagged returns, and dividend yields, while fixing the true relationship so that the model has zero explanatory power. This means that all the measured explanatory power is due to the bias.

The data generation process is set up in four simple phases:

1. We first generate a random return series by choosing returns from a normal distribution. Monthly returns are generated on the basis of a mean return of 6% annually and an annual standard deviation of 12%. Cumulative returns are calculated over the whole period as in Equation 4. For the first 11 observations of the series, the cumulative return is calculated with all the observable returns available.

2. We assume that dividends start at the value of 0.25 and grow at the constant rate of 0.3% monthly (3.6% annually).

3. From these two series we back out a price series. Remember that total returns include a price change and dividend element. We define R_t as the observed return at time t:

from which we get

The starting price is defined as the fair value in a constant growth DDM.

4. Finally, the dividend yield series is simply defined as Yld_t = Div_t /P_t.

We generate data for samples of 24, 48, 60, and 120 months. We use this data to run the two regressions:

and

We calculate R² as well as the t-statistics for both independent variables and repeat the data generation and regressions 10,000 times for each sample size.

Returns often empirically exhibit more skewness and kurtosis than is implied by the normal distribution. We take this into account by repeating the exercise assuming returns do not follow a normal but a t(5)-distribution.³

Table 2 reports the average R² obtained over 10,000 simulations. The first two columns contain the adjusted R² of Regressions 6 and 7 in the case where returns are normally distributed. The average R² is over 7% with two years of monthly data and goes down to 1.8% with ten years of monthly data. As expected, the size of the bias goes down as sample size increases, but is still nearly 2% with a comparatively long sample size of ten years. This is not negligible, especially when compared with the 2% average in-sample R² of the 60-month regressions in Table 1.

Table 2: Average adjusted R² when true R² equals zero (average of 10,000 simulations)

Our simulations help explain the difference observed in Table 1 between in- and out-of-sample statistics. Biases alone would account for R²'s of up to 8% in 24-month samples and up to 2% in ten-year samples. In other words, the in-sample R²'s reported in Table 1 should not be interpreted as evidence of true forecasting value for this TAA model but may occur in the absence of any forecasting value due to the bias described above.

In Table 1 the in-sample R² for the U.S. with a sample size of 24 months is 9.4%, while the out-of-sample R² for this model is zero. Yet the first column of Table 2 indicates that if returns follow a normal distribution, the bias should account for R²'s of "only" about 7.7 percent. Lifting the assumption of normality of returns helps explain this further discrepancy.

In the last two columns of Table 2 we drop the assumption of normality and instead assume that returns follow a t(5)-distribution. This distribution has greater kurtosis than the normal—that is to say, returns which are t(5)-distributed can be "exceptionally" large more often than in the normal distribution. The last two columns of Table 2 show higher R²'s than under the assumption of normality, particularly where the sample size is small: With 24 observations, the average R² attributed to biases goes from 7.7% to 11.1%. This may explain why the U.S. model—with an R² of 9.4% in-sample—showed no out-of-sample forecasting power in Table 1. This would confirm that the t(5)-distribution is probably a better approximation of the returns' true distribution. The difference between the normal and t-distribution results is most important with a short sample period.

Finally, Table 3 looks at biases in the t-statistics of the coefficient estimates in Equations 6 and 7. The percentage can easily be interpreted by bearing in mind that, in the absence of bias and since our returns series are randomly generated, none of the coefficient estimates in either regression should be significantly different from zero. That is to say, the t-statistics reported in Table 3 should follow the t-distribution with 2.5% probability of a statistic either above 1.96 or below -1.96.

The absence of any true relationship in our generated model means that we should expect to see significant positive coefficients 2.5% of the time and significant negative coefficients 2.5% of the time (the significance test uses a 5% rejection region). T-statistics above 1.96, however, occur nearly 33% of the time as a result of the bias for the dividend yield coefficient. This proportion does not change significantly with either the sample size or the return distribution. Measuring the predictive power of the dividend yield using in-sample t-statistics is thus fraught with problems. Again, using out-of-sample statistics will eliminate the bias.

We might expect the bias on the cumulative returns t-statistics to increase with the horizon length L considered in the cumulative returns calculation (4).⁴ Indeed, the bias is not huge with one-month cumulative returns. But with a 12-month accumulation horizon it becomes sizeable: over 9% of t-statistics indicate a significantly positive lagged return sensitivity in the shortest sample period considered. This proportion decreases to about 4.5% with a ten-year sample size, still two percent above what would occur in the absence of bias.

The in-sample bias caused by the use of lagged returns can be kept under control by insuring that the accumulation horizon is not too long and that the in-sample period is sufficiently long. With dividend yields, however, only out-of-sample statistics can eliminate the bias in t-statistics revealed in Tables 3A and 3B.

Table 3A: Average t-statistics of estimated coefficients when the true coefficients equal zero—Regression Equation (6)

Table 3B: Average t-statistics of estimated coefficients when the true coefficients equal zero—Regression Equation (7)

Summary

Investment managers undertaking TAA model-building have the choice of a wide range of statistical software packages to aid their research efforts. The new Altis System has at least two advantages over standard statistical software: Altis links the estimation procedure to the database and portfolio optimization capabilities of the World Markets Model; and it is explicitly constructed to facilitate out-of-sample testing. This allows easy application of important reliability checks. In this article, we show how this facility of Altis allows the investment manager to avoid serious in-sample biases which might otherwise seriously degrade the reliability of TAA models.

Notes