Manager Universe Measurement Problems
Survivorship Bias
Other Forms of Survivorship Bias
Management Fees
Factor Bias
Median Manager Return Analysis
Summary

Investors, consultants and managers often use performance statistics from manager universe databases to draw conclusions about the potential for success in utilizing active management in particular asset classes. Positive median manager performance relative to a passive benchmark is taken as empirical evidence that an asset class presents a particularly fertile environment for active management. For example, U.S. investors often point to international equity and U.S. smallcap equity as areas where median manager performance suggests active management can consistently add value relative to passive management. More often than not, however, it is inappropriate and, in many cases, misleading to reach these conclusions with manager universe data.

Bill Sharpe¹ points out that the average actively managed dollar must underperform the average passively managed dollar for the market as a whole or for any properly defined subset of the market:

If "active" and "passive" management styles are defined in sensible ways, it must be the case that:

(1) before costs, the return on the average actively managed dollar will equal the return on the average passively managed dollar and
(2) after costs, the return on the average actively managed dollar will be less than the return on the average passively managed dollar...

To repeat: Properly measured, the average actively managed dollar must underperform the average passively managed dollar, net of costs. Empirical analyses that appear to refute this principle are guilty of improper measurement.

If the average actively managed dollar must underperform the average passively managed dollar, then how can manager universe statistics tell a different story? There are two ways that manager universe statistics can lead to performance conclusions inconsistent with Bill Sharpe's observation that the average active dollar must underperform the average passive dollar invested:

• First, average and median return statistics associated with manager universes suffer from improper measurement and are, therefore, fundamentally flawed. Most manager universe databases suffer from some combination of omission of fees, survivorship bias, and failure to reflect persistent biases in underlying portfolios. Improper measurement implies that manager universe statistics simply do not reflect the reality of the investment experience an investor could have enjoyed.
• Second, the universe of active managers measured may represent a subset of the total universe of active managers. In this case, it is argued that the measured subset of the total universe of active managers outperforms the passive benchmark at the expense of the remaining aggregation of active assets. The remaining active portfolios - on a dollar-weighted basis - must, in this scenario, underperform the passive benchmark (before costs) by an amount equal to the outperformance of the measured dollar-weighted universe of active managers.

My primary aim in this paper is to describe and illustrate the measurement problems inherent in many manager universes, although I also comment on the notion that good relative performance in manager universes might reflect positive relative performance of a subset of active managers relative to remaining active portfolios. I conclude that it is inappropriate and misleading to use manager universe performance statistics to assess whether or not a particular asset class provides opportunity for active management. In fact, positive relative performance largely vanishes when simple adjustments are made to reflect survivorship bias, fees and factor biases. Manager universe statistics may be helpful in monitoring manager performance relative to peers (although biases can still be a problem), but provide little information to help evaluate the potential for active management. My aim is not to find fault with manager universe databases, but to point out a use of databases that is inappropriate and may lead investors to erroneously draw broad conclusions regarding asset class efficiency.

Another issue that arises in assessing asset class efficiency by observing median or average manager returns is the fact that any conclusion should be evaluated in the context of the distribution of observations that generates the observed median or average. Missing from nearly all assessments of manager universe analysis is the concept of whether one can reject, with statistical significance, the null hypothesis that manager performance is less than or equal to benchmark performance. In most cases, simply a positive median manager relative return over some investment horizon is taken as proof that an asset class is inefficient and fertile for active management.

Investors should not entertain the hypothesis that a sub-group of measured active managers outperforms a sub-group of remaining active managers until and unless the manager universe data are free of the measurement problems we outline. Our assessment of manager universe data suggests that the measurement problems are so significant and pervasive that the argument that a subset consistently outperforms is largely moot. When adjustments are made to reflect measurement problems, we find that in asset classes where managers are thought to add value managers, in fact, do not add value.

Bill Sharpe is right. The average actively managed dollar does underperform the average passively managed dollar. The question for investors contemplating active management is not a matter of whether a particular asset class provides a better environment for active management. The question is whether a particular manager will add enough value relative to a passive benchmark to overcome higher costs and increased risks. For investors in aggregate we know the correct answer is 'no' - active managers do not add value.

Manager Universe Measurement Problems
Manager universe databases suffer from three fundamental forms of mismeasurement:

1. Survivorship Bias
2. Omission of Fees
3. Factor Bias

Mismeasurement related to survivorship bias and omission of fees each tend to bias average or median return statistics higher, while factor biases can bias return statistics in either direction. We find that basic efforts to adjust average or median return statistics to account for improper measurement generate returns statistics far different from unadjusted return statistics. Properly adjusted return statistics can lead to very different conclusions than those reached by casual observation of unadjusted manager universe return statistics.

To illustrate the extent to which casual interpretation of median manager statistics can generate misleading conclusions, we analyze the median manager returns of international equity managers from the Callan manager universe. We used the Callan database because of its lengthy history and solid reputation among users. Our Callan universe data included quarterly median manager returns and median manager returns for holding periods from 5 years to 25 years (in 5-year increments). All holding period returns are for the period ended December 31, 2001. As is often the case, manager-level return information was not available, so we were unable to carry out tests of statistical significance of median excess return observations. We analyzed international equity managers because of the commonly held belief is that active managers consistently add value in the international equity category. Later in the paper I summarize similar analysis for Callan's U.S. Large Cap and U.S. Small Cap manager universes.

Investors point to statistics like median manager returns relative to a benchmark (Table 1) as proof positive that, on average, international equity managers add value relative to the passive investment alternative.

On the surface this seems a very convincing argument. In all holding periods the median active international equity manager returns in the Callan universe exceed the benchmark MSCI EAFE return by anywhere from 1.2% annually to over 3% annually. Often the analysis stops here. However, after adjusting for basic flaws in the data the conclusions change dramatically.

Survivorship Bias
Survivorship bias comes in many forms. The most prevalent form occurs whenever one observes holding period returns. The bias occurs because holding period returns represent the performance of the subset of managers that "survived" the holding period, not the full opportunity set of managers that existed at any point in time during the period. Although the importance of survivorship bias is well documented in studies of, for example, mutual fund performance (see Carhart, Wermers or Davis²). Figure 1 illustrates the number of international equity managers in the Callan universe reporting returns at the beginning of the period versus those who reported returns for the full period. For most holding periods, the number of surviving firms is far lower than the number reporting returns at the beginning of the period. For example, for the 20-year holding period only 31 of the 69 firms reporting returns at the beginning of the 20-year period survived the full period. (Curiously, for the 5-year period there are more holding period returns than there were managers reporting returns at the beginning of the period. More on this suspicious outcome shortly.)

It is not surprising that the longer the holding period the fewer the number firms that report returns for the full holding period. A bias in returns occurs because, on average, the managers that survive the full period are more likely to be those who generated above-average performance. Winners typically live to manage another day, while losers often times do not. One way to adjust for the presence of this form of survivorship bias is to evaluate the median returns observed in any given quarter rather than the median holding period returns. The median returns observed in any give quarter represent the median return of the full opportunity set of managers (survivors and non-survivors) available to investors at any point in time. Figure 2 illustrates how consistently positive and significant this form of survivorship bias is for the Callan international equity manager universe. Using the 20-year median return as an example, the excess return of the median manager fell by over 50% from 1.91% for holding period median returns to 0.92% for quarterly median returns. Survivorship bias in holding period returns is material and, not surprisingly, consistently biases median returns upward. For all periods, survivorship bias generates holding period median returns significantly higher than the median returns associated with the opportunity set of managers that were truly available to investors.

Survivorship bias inherent in holding period returns does not imply there is a problem with the way a manager universe database is built or maintained. Holding period return survivorship bias surfaces as an issue because manager universe data is arranged and interpreted in an inappropriate manner. Analyzing holding period returns, in effect, assigns the positive bias associated with surviving firms as an attribute of the asset class rather than the positive bias it is.

Other Forms of Survivorship Bias
Survivorship bias associated with holding period returns occurs as a natural result of a limited number of firms surviving a given observed holding period. Another form of survivorship bias relates to how data survives in, or is added to, databases. This type of survivorship bias - sometimes referred to as inclusion bias - occurs when those responsible for maintaining databases remove and/or add manager return observations in a manager universe database.

David Swensen, Chief Investment Officer of Yale University's Endowment, illustrates in his book Pioneering Portfolio Management the potential for inclusion survivorship bias with an example from the PIPER Managed Accounts Report. The median international equity return for 1992 reported following year-end by PIPER was -6.4%, and represented a sample of 120 managers. Oddly enough, PIPER's assessment of the median 1992 international equity manager return changes as the vantage point rolls forward in time. Table 2 shows the median international equity manager return (and number of managers reporting a return) for 1992 as reported in 1992, 1993, 1994, 1995, 1996 and 1997. The table illustrates that the 1992 median international equity manager return rises 2.3 percentage points (-6.4% to -4.1%) by the time we reach 1997, and the number of managers rises over 60% (from 120 in 1992 to 194 in 1997).

Source: Swensen

So the manager with median performance in 1992 finds that he is a mid-3^rd quartile manager for 1992 when viewed from 1997! Clearly the PIPER international equity manager universe has grown over time and, because the added managers are biased toward better performing managers, the median return drifts substantially higher over time. The old adage that "history is written by the winners" seems to hold true in the case of some manager databases. Swensen finds this inclusion bias consistently biases returns upward across asset classes. While we do not have the historical data to assess whether inclusion bias influences the Callan international manager database, the fact that there are more 5-year holding period manager returns than there were beginning-of-period returns suggests that the database may reflect return data that has been added over time. Our efforts to correct for holding period survivorship bias do not address the potential for inclusion survivorship bias in the Callan international equity manager database.

Management Fees
An obvious, but often overlooked measurement problem with manager universe statistics is the lack of adjustment in median manager returns to reflect the average level of management fees. Since management fees vary across clients for a number of different reasons, manager universe statistics typically show manager returns before fees. According to a recent Greenwich Associates study, the average institutional management fee paid to international active managers was 69bp. Figure 3 shows how the Callan median international equity returns fall after adjusting for both holding period survivorship bias (although we do not adjust for the inclusion effect) and management fees. Again using the 20-year holding period returns as an example, what appeared to be 1.91% in annualized median manager excess return falls to 0.24% excess return after simple adjustment for holding period survivorship bias and management fees. What appeared to be a powerful argument in favor of active international equity investing becomes much less convincing after simply making adjustments to reflect the most common form of survivorship bias and average management fees. Recall also that this adjusted median manager return - even if it is a positive number - says very little if it is not evaluated in the context of the distribution of manager returns that generates it. Demanding statistical significance holds evaluation of asset class efficiency to a significantly higher standard.

Factor Bias
The third and final form of manager universe mismeasurement is the extent to which median manager returns reflect persistent factor biases. If evidence suggests median returns are a product of consistent factor biases relative to the benchmark portfolio, then, in effect, the wrong benchmark is being used. It is important to note that we are not suggesting that changing factor exposures - or "bets" - that reflect managers' insight into attractiveness of groups of securities at particular points in time are a form of measurement error. However, if median manager returns reflect permanent tendencies to be overweight or underweight factor exposures of a benchmark used to gauge relative performance, then investors should be careful not to confuse incremental return associated with factor exposures with value-added. Unlike survivorship bias and omission of management fees, factor biases can generate median manager returns that are misleadingly high or low.

To assess the potential for factor biases in the Callan international equity median manager returns, we used regression analysis to attempt to explain excess median manager returns. Only the excess return that is not explained by persistent factor exposures can then attributed to aggregate skill in the manager universe. In the regression, we use the excess median manager returns that are adjusted for holding period survivorship bias and fees. We then regress those quarterly excess median manager returns on three commonly cited variables (factors) that might be helpful in explaining excess international equity manager returns: (1) The weight of Japan relative to other countries, (2) The presence of cash holdings (or leverage), and (3) The presence of currency hedge (to US$) positions.

The regression results are shown in Table 3. Surprisingly, each of the proposed variables has statistically significant explanatory power in describing the median manager excess returns. The intuitive interpretation of the regression results is that the difference between observed median manager returns and benchmark returns is largely explained by a persistent bias to (1) be underweight Japan, (2) hold cash, and (3) maintain a partially hedge of FX exposure. Again, we are not characterizing changing active positions reflected through the proposed factors, but are capturing ongoing biases reflected in the full 28 years observed in the sample of quarterly median manager returns. For example, the persistent bias apparent in the median manager excess returns for managers to be underweight Japan implies that managers were underweight Japan both in periods of relative strength and weakness in the Japanese equity market. Breaking the regression analysis into sub-periods confirms the signs of coefficients generated in the regression analysis are unchanged. In the latter half of the sample period this bias translated into significant positive incremental median manager return, but in the first half the bias generated significant negative incremental return. The important point is that a persistent bias does not represent value-added attributable to manager skill, and median manager returns should not be interpreted as such.

The intercept of the regression equation - the quarterly stock selection alpha - is modestly negative (-0.05%), which implies the annual alpha provided by the median manager return after adjusting for factor biases, management fees and survivorship bias is on the order of -0.20%. The difference between the naïve assessment of international active manager median returns (1.2%-3.0% annual excess return) and analysis that adjusts for common mismeasurement problem (-0.20% annual excess return) suggests how significant and misleading unadjusted manager universe statistics can be.

Median Manager Return Analysis for Other Asset Classes
We also adjusted median manager returns of Callan's US large cap and US small cap manager databases to reflect holding period survivorship bias, management fees and factor biases. The results were consistent with what we found for Callan's international equity manager database. Our results in the small cap arena are consistent with those of Richard Ennis and Mike Sebastian in their paper titled The Small-Cap Alpha Myth³. As with the international equity manager database, holding period survivorship bias consistently biased returns upward. Figures 4 and 5 illustrate how median manager returns for Callan's small cap and large cap manager universes change when adjusted for holding period survivorship bias and management fees. As with the international equity manager database, adjusted median manager returns decline significantly for both the small cap and large cap manager universes.

Factor exposures also play a significant role in explaining excess median manager returns for both small cap and large cap manager universes. Table 4 and Table 5 shows the outcome of regression analysis that assesses how excess median manager returns are the product of factor exposures. For both small cap and large cap manager universes we tested whether three commonly cited variables - (1) relative return of large cap vs. small cap, (2) relative return of benchmark vs. T-bills, and (3) relative return of growth vs. value - explain adjusted median manager alphas. Small cap median manager excess returns are largely explained by a persistent bias to be overweight the large cap vs. small cap factor and by a beta greater than the benchmark. Large cap median manager excess returns are in turn explained by a persistent bias to be underweight the large cap vs. small cap factor and to maintain cash balance or below-benchmark beta. In neither small cap nor large cap regressions did the growth/value relative return factor explain excess returns. As was the case with the international median manager excess returns, unexplained median manager alpha was modestly negative for both the large cap and small cap Callan manager universes.

Summary
Manager universe databases are of little use in making determinations about asset class efficiency. The illustrations we have provided show how significant and positively skewed the survivorship, management fee and factor biases are for a typical manager universe. After simple adjustments for the most common forms of bias, the median manager statistics for each of three manager universes I evaluated are entirely consistent with the notion that the average active manager underperforms its appropriate benchmark due to the presence of transaction costs and management fees. Even the two asset classes (international equity and U.S. small cap) that are often held up as examples of arenas where active managers can consistently add value lose their active management luster when appropriate adjustments are made to the median manager data. One benefit for those assessing these asset classes is that there is no longer any need to try to explain how the observed universe of managers is the better-performing subset of active managers. With adjustments for common biases, the universes look suspiciously like their respective benchmarks.

1 William F. Sharpe, "The Arithmetic of Active Management," The Financial Analysts' Journal Vol. 47, No. 1, January/February 1991

2 Mark Carhart, "On Persistence In Mutual Fund Performance," Journal of Finance, March 1997. Russ Wermers, Journal of Finance, August 2000. Jim L. Davis, "Mutual Fund Performance and Manager Style," Financial Analysts Journal, January/February 2001

3 Richard M. Ennis and Michael D. Sebastian, "The Small-Cap Alpha Myth," Journal of Portfolio Management, Spring 2002.

This material is for your private information. The views expressed are the views of Eric Brandhorst only through the period ended November 22, 2002 and are subject to change based on market and other conditions. The opinions expressed may differ from those with different investment philosophies. The information we provide does not constitute investment advice and it should not be relied on as such. It should not be considered a solicitation to buy or an offer to sell a security. It does not take into account any investor's particular investment objectives, strategies, tax status or investment horizon. We encourage you to consult your tax or financial advisor. All material has been obtained from sources believed to be reliable, but its accuracy is not guaranteed. There is no representation nor warranty as to the current accuracy of, nor liability for, decisions based on such information. Past performance is no guarantee of future results.

Posted On: December 04, 2002