If ESG ratings leave you confused, you’re not alone. In general, environmental, social, and governance investing is mired in inconsistencies, with the funds and indexes having vastly different approaches. While ESG ratings aim to help investors, they often just confuse the matter further. Researchers associated with the MIT Sloan School of Management’s Sustainability Initiative have been studying what they have dubbed the “aggregate confusion” of the information included in ESG ratings. In fact, the research team calls itself The Aggregate Confusion Project.
In April, the Project published its latest paper, “Aggregate Confusion: The Divergence of ESG Ratings.” The researchers looked at the divergence of ESG ratings based on data from six ESG ratings agencies: KLD, Sustainalytics, Moody’s ESG, S&P Global, Refinitiv, and MSCI. What the researchers found, unsurprisingly, was that different agencies “disagree substantially” in their ratings.
“ESG ratings have become an immensely lucrative pseudoscience,” said Bryan Junus, Chief Analyst for The Corporate Citizenship Project, an Iowa-based think-tank taking a data-driven approach to corporate governance matters. “Given that most ESG ratings are primarily based on arbitrary standards or the result of companies ‘gaming’ the ESG rating system, it stands to reason that ESG ratings would be widely divergent.”
Correlations among ESG ratings ranged between 0.38 and 0.71. Correlations are scored on a scale of 0 to 1, where 0 indicates no correlation and 1 indicates perfect correlation. Credit rating agencies, to cite another common ratings area, show a correlation of 0.92, far higher and much less noisy than ESG ratings correlations.
Researchers identified three types of divergence: scope, measurement, and weight. Scope divergence refers to agencies’ rating based on different sets of attributes – for example, some agencies include lobbying activities while others do not. Measurement divergence refers to agencies measuring the same attribute using different indicators, such as measuring labor practices by turnover or court cases. Finally, weight divergence refers to the relative importance of each attribute in a particular agency’s rating. These divergences “are intertwined, making it difficult to interpret the difference between two ESG ratings,” the researchers note.
In order to tease apart the effects of the divergences, the researchers categorized 709 ESG indicators into 64 categories used by the six rating agencies. Then they calculated category scores for each rater, ran regression models to obtain and compare fitted ratings, and finally decomposed the divergence into the contributions of scope, measurement, and weight.
The researchers found that they could estimate with 79% to 99% accuracy the agencies’ implied aggregation rules using those categories. They conclude that “although ESG ratings have incompatible structures, it is possible to fit them into a consistent framework that reveals in detail how much and for what reason ratings differ.”
A second interesting finding is that “measurement divergence is the main driver of rating divergence, contributing 56% of the divergence. Scope divergence is also important, contributing 38%, while weight divergence contributes a mere 6%.”
Finally, there is also a rater effect, or the so-called “halo effect,” where a company receiving a high score in one category would likely receive high scores in all other categories. According to the study, “the rater effect is substantial … [and] explains 15% of the variation of category scores.”
That halo effect shines a light on the way “patterns influence how firms are assessed” for ESG scores. The researchers do not “conclusively identify” a cause for the halo effect, but they do suggest that a “promising avenue for future research is to investigate additional reasons why ESG ratings might deviate systematically in their assessment, for example, whether economic incentives to adjust ratings exist.”
Junus agreed mostly with the conclusions of the study but says there is a significant data point missing.
“We find the MIT study very on point in its assessment of divergence in ESG ratings. However, we think they also have to consider the fact that companies like ISS and S&P have potential significant conflicts of interest that are responsible for these divergences. These conflicts of interest are equally significant but different. ISS, for example, has a conflict of interest because it provides both ESG ratings and also provides ‘ESG consulting’ to the same companies it rates. The appropriate follow-up study to this one would be to analyze the correlation of ESG ratings with payments to ESG consultants and ratings firms including ISS.”
Rashida Salahuddin, President of The Corporate Citizenship Project, cautioned investors from trusting the “pseudo-science” of ESG ratings.
“Miriam Webster Dictionary defines pseudo science as ‘a system of theories, assumptions, and methods erroneously regarded as scientific’. Pseudo-science has long been a way for powerful companies to advance their agendas at the expense of ordinary people. Unless ISS and other ratings agencies turn over their methodology and can show their ratings are based on quantitative standards, we have no choice but to assume these ratings are yet another deceptive pseudo-science.”