Most of the time, past results don’t change. Sometimes they do, no matter what data set you use. Here we go over a common cause of when they do.
Why would a stock (say Google) be included in a screener as of October but not when you backtest during November? Here we explain how ranks, which is used in almost every Equities Lab score, can be vulnerable to change despite Equities Lab’s use of Morningstar’s world class bias free data.
The fundamental problem is that a stock’s rank can be sensitive to every other stock in the system. Morningstar Data tries to be stable and perfect, but as Janet in the Good Place says, “Pobody’s Nerfect!” When Morningstar’s data team realizes there was a data entry error they must fix it. The problem is, fixing that single stock can change the rankings of every other stock in the system.
You could try ranking over only large caps as the vast majority of time the data errors lay in small or micro cap stocks. Those rankings though, may not be as robust as ones that use more data and that score would only be useful for large cap stocks. You can also try avoiding using rankings. For example Piotrowski P score does not use any rankings, but you will lose access to some very useful metrics if you don’t permit yourself to rank.
What’s worse is that almost all built-in scores rank across all reasonably sized stocks, as the Equities Lab data team can’t assume that users will only want to trade large caps, so scores are built accordingly.
What’s worse is that almost all built-in scores rank across all reasonably sized stocks, as the Equities Lab data team can’t assume that users will only want to trade large caps, so scores are built accordingly.
Finally, we have the problem that when a company’s industry is changed, we can’t tell if it was changed because the company was misclassified or if the company changed directions. As about 80% of the time, the company wasn’t in the correct industry to begin with, that’s the decision we go with, so companies that change their classification will also change your results.
The brutal truth is that if we accept any error corrections from Morningstar some ranking results will change. Morningstar’s data is remarkably good and robust, so this effect should be very small and not come up that often. Nonetheless, we are writing this article because it’s a frequent cause of support questions, so we wanted to cover it in depth here.