Newsletter #168 Home
Newsletter Contributors
Previous Issues
BARRA Home




Developing and
Implementing
Risk Management
Systems

Fortis Group
Case Study



American Depository
Receipts in The
Global Equity Model



Analyzing the
Performance of
Crossing Networks
The Market
Impact Model™ — Part 4



The BARRA Brainteaser
 for Fall 1998
Solution to the Summer
1998 Brainteaser

The Market Impact Model,
Part 4: Testing the Market Impact Model

by Nicolo G. Torre and Mark Ferrari

In previous articles in this series, we have described the purpose of the Market Impact Model, the economic intuition behind it and the model itself. Now it is time to explore how well the model actually works. There is a wide variety of tests that may be made of the model. We shall begin with the simplest test and work up to the more complicated ones, sharpening our understanding of cost modeling as we go.

The simplest test one might think to make of the Market Impact Model is to forecast the cost of a particular trade, then work the trade and compare the resulting cost to the model's forecast. Unfortunately, this test is not particularly informative because the model forecasts the expected or average cost of trading. Any one trade, however, may happen to arrive in the market simultaneously with a large competing or complementary order. Consequently, the cost experience for a single trade will depart, often significantly, from the mean experience. Testing the Market Impact Model against an individual trade lacks power, for the same reason that a valuation model cannot be evaluated by looking at the performance of one name on its buy list; little is learned by comparing the mean of a distribution with just one draw from that distribution.

The evident solution to this problem is to look at a larger number of trades. Comparing the mean cost of a sample of similar trades with the mean forecast by the model increases the power of the test. The comparison can be made for a number of different samples of different types of trades. Estimating the mean cost of the executed trades in each sample is itself an interesting problem. We will return to this momentarily, but for now, let us suppose that we have only a rough notion of our trade costs, such as the trading desk's impression of what it costs to do the trades. From such rough estimates we will not be able to develop any very precise measurement of model performance. However, we can still conduct some basic validation of the model. We might begin by sorting a sample of trades into two groups. These groups can be distinguished in various ways—for instance, the date on which the trade occurred, the type of assets traded, buys or sells, how the trades were worked, trade sizes, and so forth. We could choose the groups to be identical with respect to all characteristics save one. For instance, we might take one group to be trades in large capitalization stocks and the other to be trades in small capitalization stocks. For the costs of each group we will have only our rough estimate, but as long as the error in our estimate is independent of the dimension in which the groups differ, we should have a reasonable measurement of their relative cost to trade (e.g., the small capitalization stocks are three times more expensive to trade, all other things being equal). Next, using the Market Impact Model, we can forecast the cost to trade both groups and by taking the ratio arrive at a forecast of the relative cost. Comparing the forecast of relative cost to the estimated realized relative cost then provides a test of model performance along the dimension that distinguishes the two groups. Performing tests of this sort along multiple dimensions allows one to build a basic confidence that the model is behaving reasonably.

Next, one would like to move towards a more quantitative measurement of model performance. For this purpose, one requires a good measure of realized cost. A conventional choice is to measure cost as the difference between the price paid and a pre-trade reference price. It must be noted, however, that the number that results is not a pure measure of cost because it includes the asset's return due to factors other than one's own trading activity. For large orders worked over a long time horizon, such factors may dominate the measured value. Thus, it must be realized that costs are not directly observed, but rather are estimated with a varying level of measurement error. When applied to data containing measured error, many standard techniques (such as regression analysis) can lead to erroneous conclusions. Fortunately, the application of appropriate statistical techniques can compensate for these measurement errors.

The first step in analyzing data containing measurement error is of course to reduce the error as far as possible. A major source of error in estimating cost as the difference between a pre-trade reference price and the trade price is the confounding effect of other's trading. However, for the purpose of testing the Market Impact Model, there is no necessity to limit our analysis to trades originated by a single investor. Instead, one could aggregate trades placed by several investors and look at the cumulative impact. The natural extension of this idea is to aggregate all trading which occurs in a time interval. By doing so, we may eliminate the confounding effect of trading by others. We have found a half-hour period to be a suitable aggregation interval as it balances the need for multiple measurements during a trading day with the need for each measurement to be based on a reasonable level of activity.

The second step in analyzing the data is to summarize the measurements in a form suitable for comparison with the forecast. The forecast relationship between impact and volume is very nearly a square-root relationship. Accordingly, we summarize the data by fitting the curve:

Cost =

to the measured data. Similarly, we fit the curve

Forecast =

to the model forecast. Then comparing g_emp to g_mdl gives an evaluation of model performance.

The third step in our analysis is to take a quick look at the data to see the basic relationship. For each stock i on day t we have a model forecast

and a measured result

.

To summarize the data, we bin these observations into one of 20 bins on the basis of

.

Let and

denote the averages of

and

respectively over the nth bin. FIGURE 1 then shows a graph of

versus.

FIGURE 1: Comparison of aggregated forecasts and measurements on log-log axes

As can be seen there is a clear linear relationship between the mean of the forecast and the mean of the observation. It can be also observed, however, that the measured cost is generally less than the forecast cost. To understand this discrepancy between forecast and realization, we must probe the data more extensively.

There are essentially two hypotheses that might explain the discrepancy. One is the possibility of systematic error in the model. Were this to be the case, one could solve the problem simply by scaling the model forecast down. The other possibility is a systematic error in the measured cost, which results in the measurement underreporting the true cost. Clearly it is important from an investment viewpoint to distinguish between these hypotheses. We would not want to adjust the model to an erroneously low estimate of cost and thus stimulate investors to overtrade and so incur the actually higher true cost.

To distinguish between model error and measurement error, we must introduce a third estimate of transaction costs to arbitrate between the other two. Technically this third estimate is known as an instrumental variable. Consider the situation where we have three estimates of the transaction cost: the model estimate g_mdl, the empirical measurement g_emp, and the instrumental estimate g_inst. Each of these estimates is related to the true cost g_true by

(1)

The departure of the coefficients c_mdl, c_emp, and c_inst from 1.0 captures the systematic bias in the estimates and the variables e_mdl, e_emp, and e_inst represent random (unsystematic) errors. Although the true value g_true is unobserved, one can still solve the system of equations to find

provided the random errors are uncorrelated. To see how this is possible, consider the computation of the covariances implied by Eq. 1. For instance

This equation relates an observable quantity on the left hand side to three unobservable quantities on the right hand side. Calculating all the other observable covariances in this manner produces a set of equations which could in principle be solved to find the unknown coefficients

.

In practice, one deals with finite data samples. Thus one does not solve the equations exactly, but rather finds the statistical estimates

which are most consistent with the Eq. 1 and the actual data.

To apply this methodology, we require an instrumental variable. The method will work even if the variable is not a particularly good estimate of the true cost, thus we need not assume that c_inst = 1 or that var(e_inst) is small. However, we do require that the errors e_inst in the instrument be uncorrelated with the errors e_mdl and e_emp in the other two estimates. For an instrumental variable we use the square root of the ratio of trade size to capitalization. Thus, the assumption is that a trade of fixed size will cost more in a small cap stock than in a large cap stock. This assumption is intuitively appealing and is borne out by actual trading experience. In the construction of the Market Impact Model we do not use capitalization as a variable, so it is reasonable to expect that the model errors will be uncorrelated with the errors in the capitalization instrument. Similarly, we can expect the observational errors in the empirical estimate to be uncorrelated with the capitalization estimate. Hence, capitalization can serve as a suitable instrumental variable for our analysis.

Applying this methodology we can decide between the hypothesis of systematic error in the model or in the measurement. The full details are presented in the Market Impact Model Handbook, so here we shall just summarize the results. There is no evidence that the Market Impact Model systematically overpredicts transaction costs. If anything, the data are consistent with a slight underprediction (approximately 6%). On the other hand, there is compelling evidence that the empirical estimate is systematically biased low, by about 35%. This is a very interesting result! Intuitively we understand how this bias could arise. Survivorship bias deletes the expensive trades from our sample, because investors will not trade in the face of excessive costs. Until now we had no way of knowing whether this bias was large enough to matter. Now we have concrete evidence that the effect is real and significant. In particular, investors who extrapolate from past trading experience to forecast their future costs are setting themselves up for a significant unpleasant surprise. Their actual costs will prove higher because past executed trades represent a biased sample of market conditions, and biased samples produce inaccurate forecasts. In contrast, forecasts based on the Market Impact Model have proven to be largely free of systematic errors.

Having verified that the Market Impact Model is free of systematic error, we next turn our attention to estimating the size of the random error. The answer is simply the variance of emdl. On its own, however, this quantity is not very interpretable. We seek, therefore, a figure of merit that speaks to the practical usefulness of the model. We take as our standard of accuracy the cost estimate for a list of trades in fifty randomly selected stocks. The attraction of this standard is that it corresponds to the application for which the model was designed, namely evaluating the cost-benefit tradeoff of a portfolio rebalancing. Based on our data analysis, we find that the median error in the model cost estimate for such a list is generally less than 10%. To put this level of performance in perspective, this is the same accuracy we estimate for BARRA risk models applied to similar sized portfolios. In retrospect, perhaps it is not surprising that the performance of the Market Impact Model should turn out comparable to the risk model, because the Market Impact Model itself has a risk model at its heart.

In this series of articles covering the Market Impact Model, we began by describing the goal we set ourselves, then we went on to describe our attack on the problem and how we have verified attainment of our goals. Yet the justification for all of this effort must be not in the thing itself, but rather in its usefulness for practical portfolio management. It is to the applications of the model that we shall turn in our next installment in this series.