Rating a Robo-Rater
David Nanigian – CSUF Mihaylo School of Business
-- In September 2011, Morningstar, Inc., launched the Morningstar Analyst Ratings to help investors select mutual funds. Unlike the Morningstar Star Ratings, which assign a grade to past performance, the Morningstar Analyst Ratings are designed to provide investors with an assessment of a fund’s ability to outperform. The process employed by Morningstar in assigning Morningstar Analyst Ratings is rather labor-intensive. Therefore, it is not feasible for Morningstar to assign a Morningstar Analyst Rating to every mutual fund.
In an attempt to overcome this challenge, as I explain in my recent paper, "Rating a Robo-Rater," Morningstar developed the Morningstar Quantitative Ratings to provide an assessment of a fund’s suitability for investment for those funds without a Morningstar Analyst Rating. The Morningstar Quantitative Ratings are generated by a machine-learning model that is designed to emulate the decision-making processes of the analysts in Morningstar’s Manager Research Department that assign the Morningstar Analyst Ratings. With the introduction of the Morningstar Quantitative Rating, the number of open-end united states mutual fund portfolios with a forward-looking rating by Morningstar increased from 1,366 in May 2017 to 7,760 in June 2017. Since June 2017, the market for mutual funds contains both funds with Morningstar Analyst Ratings and funds with Morningstar Quantitative Ratings. This provides a laboratory to gauge the value of “soft information” in analyzing mutual funds.
To gauge the value of “soft information” in analyzing mutual funds, actively managed U.S. equity mutual funds from Morningstar Direct’s survivor-bias-free United States Mutual Funds database are sorted into portfolios based on their Morningstar Quantitative Rating or Morningstar Analyst Rating assignment in the prior month. The performance of the portfolios of funds over the time period from July 2017 to June 2019 is then evaluated based on the alpha from Carhart’s Four-Factor Model, which augments the classical Capital Asset Pricing Model with three additional variables to control for potential size, value, and momentum factors in stock returns.
The below figure shows the relationship between Morningstar Analyst Rating and performance and also the relationship between Morningstar Quantitative Rating and performance using equally-weighted portfolios of funds. The relationship between Morningstar Quantitative Rating and performance is rather weak and unstable, with the portfolio of funds with gold ratings outperforming the portfolio of funds with negative ratings by 0.90 percentage point per year. In contrast, the relationship between Morningstar Analyst Rating and performance is strong and nearly monotonic, with the portfolio of funds with gold ratings outperforming the portfolio of funds with negative ratings by 2.48 percentage points per year. These results indicate that “soft information”, which is factored into the assignment of Morningstar Analyst Ratings but not Morningstar Quantitative Ratings, is indeed valuable in mutual fund selection.
Next, the analysis is rerun using value-weighted portfolios of funds (weights are proportionate to total net assets). The figure below shows that there is no apparent relationship between Morningstar Quantitative Rating and performance. It also shows that there is a strong and positive relationship between Morningstar Analyst Rating and performance, with the portfolio of funds with gold ratings outperforming the portfolio of funds with negative ratings by 6.21 percentage points per year. This is 19 times the magnitude of the performance differential between the portfolio of funds with gold Morningstar Quantitative Ratings and the portfolio of funds with negative Morningstar Quantitative Ratings. This corroborates the results involving the equally-weighted portfolios and further illuminates that “soft information” is incredibly valuable in mutual fund selection.
The analysis was also rerun using gross returns rather than traditional net-of-expense returns, which provided additional evidence of the value of “soft information” in mutual fund selection. To learn about these additional results and an explanation for why there is a positive (albeit weak) relationship between Morningstar Quantitative Rating and performance, read my working paper, “Rating a Robo-Rater”.