The past two decades have seen an explosion in organisations’ use of increasingly sophisticated predictive analytics models to improve forecasting and planning. The extensive availability of big data on customer, market, and competitor information, paired with lower data storage costs and faster data processing, has accelerated this trend. Since these models are less susceptible to the biases and organisational politics that can often undermine managerial forecasts, organisations face a critical question: Is human involvement necessary when making decisions with predictive analytics?
The research study
The shortcomings of human judgement in forecasting are well known. People tend to be overly optimistic or ignore information that does not fit their preconceived notions. Managers often use shortcuts and rules of thumb that are ineffective.
However, predictive analytics models have their own set of limitations. They have poor predictive capabilities in cases of substantial uncertainty or when historical data is limited. They are also typically slow to reflect changing conditions.
These drawbacks present an opportunity for managers to seek out information that is not or cannot be digitised and use their expertise and intuition to improve upon the predictive model’s recommendations.
Knowing when to incorporate manager discretion in predictive analytics is critical for organisations as they seek to improve forecasting performance with limited resources. Our CIMA-sponsored research, “(When) Does Human Intervention in Predictive Analytics Judgements Help or Hurt? A Management Control Perspective”, provides new evidence on when having a person in the loop is more beneficial than using predictive analytics models on their own (see the research executive summary).
Research approach
This study utilises proprietary data from an automotive parts retailer to examine product assortment planning decisions where a predictive analytics model and manager judgement jointly determine which products to stock at which retail locations.
The retailer supplies nearly a halfmillion unique parts, but an average retail store can only stock about 25,000. The predictive analytics model identifies parts that are most likely to sell based on a large and varied dataset related to product and store attributes. The model is used to make stocking decisions for thousands of retail locations. As in many settings, a handful of managers at this company have the authority to override the product stocking recommendations.
What we found
Our study of one product category found that managers override the predictive analytics model’s product stocking recommendation about 48% of the time. And in 53% of all decisions, managers make the “right” decision — that is, they follow the model’s recommendation when justified (they either make no override or override the model in the appropriate direction).
However, managers are not perfect. For about 10% of stocking decisions, we found that managers fail to intervene when they possibly could have improved on the model’s recommendation, and, in 38% of cases, they override the model when they likely shouldn’t have.
Manager judgement appears more valuable, though, when managers override the model’s recommendation based on information and insights not contained within the historical data that the model uses. This may occur when historical data is limited, as with new products. For example, the odds a manager will override a model recommendation are 10% higher for a brand new product than for a part that is five years old.
Manager insights also add value when environmental uncertainty is greater, such as when there is higher volatility in expected demand. When the model identifies parts that have high demand volatility, the odds are 5.71% higher that managers will override the recommendation as compared to parts with low volatility. The study finds that manager discretion improves stocking decisions in these instances because managers anticipate changes in probability the product will sell better than the model.
These findings suggest that managers can capitalise on the weaknesses of predictive analytics models, even when they have no explicit indicators of the model’s performance. They use their expertise and intuition to effectively choose when to override the model, and their decisions result in superior forecasting decisions (see the chart “When Managers Should Override Stocking Decisions,” below).
When managers should override stocking decisions
Impact of stakeholder pressure
Though managers have discretion to override the model, their decisions are not made in a vacuum. These decisions are informed by the political pressures and conflicts of interest inherent in any organisational context.
Our analysis shows the impact of these external pressures. For example, managers faced with pressure from key external stakeholders with misaligned incentives (independent store owners) tend to accommodate stakeholder preferences at the expense of forecasting quality. Managers report that independent store owners appear more cash conscious and view the predictive analytics model as a sales ploy rather than a tool to improve assortment planning.
According to our results, managers are no more likely to override model stocking recommendations for independent stores than they are for company-owned stores. However, when managers override the model for company stores, they do so downward only 7% of the time. When managers override the model for independent stores, however, they override downward 37% of the time, in line with independent store owner preferences. This means the manager is more likely not to stock a product that the model recommended to stock. In these situations, manager discretion results in worse stocking decisions because managers are often choosing not to stock a product that likely would have sold.
Improving forecasting performance
The study reveals that organisations can improve forecasting performance by incorporating manager discretion, especially in cases when predictive analytics models have limited historical data or there is greater environmental uncertainty.
While human judgement can improve decision-making over predictive analytics alone, that improvement can be undermined by other people in the process with competing interests and incentives. That is, while managers can outperform sophisticated models in certain circumstances, they are also susceptible to institutional pressures stemming from misaligned incentives across the organisation.
To ensure high-quality forecasting, organisations can take steps to limit the influence of stakeholders with preferences that may be at odds with the organisation’s objectives. They may also put control mechanisms into place (for example, contractual provisions like selling on consignment) to better align stakeholders’ incentives with those of the organisation.
Organisations face increasing pressure to curb manager discretion in favour of data-driven decision-making across various management tasks including demand forecasting, inventory planning, budgeting, and performance evaluation. Our study, however, highlights the value of human judgement in the process, especially when the management control systems support decision quality.
Jen Choi, Ph.D., is an assistant professor of accounting at the University of Michigan in the US; Ewelina Forker is an assistant professor of accounting and information systems at the University of Wisconsin-Madison in the US; Isabella Grabner, Ph.D., is a professor at the Institute for Strategy and Managerial Accounting at WU Vienna in Austria; and Karen Sedatole, Ph.D., is the Asa Griggs Candler Professor of Accounting at Emory University in the US. To comment on this article or to suggest an idea for another article, contact Oliver Rowe at Oliver.Rowe@aicpa-cima.com.
AICPA & CIMA RESOURCES
“4 Ways CFOs Can Maximise the Benefits of Predictive Analytics”, FM magazine, 24 May 2023
“How Algorithms and Human Conversations Can Remove Forecasts’ Bias Perception”, FM magazine, 26 July 2021
LEARNING RESOURCES
CQ Originals: Critical Thinking
Become a critical thinker and pave the way for improved problem-solving communication and collaboration skills.
COURSE
This product teaches valuable analytical tools for managing risks and making informed decisions.
COURSE
