Skip to main content

Creating Simple Rules for Complex Decisions

Machines can now beat humans at complex tasks that seem tailored to the strengths of the human mind, including poker, the game of Go, and visual recognition. Yet for many high-stakes decisions that are natural candidates for automated reasoning, like doctors diagnosing patients and judges setting bail, experts often favor experience and intuition over data and statistics. This reluctance to adopt formal statistical methods makes sense: Machine learning systems are difficult to design, apply, and understand. But eschewing advances in artificial intelligence can be costly.
Recognizing the real-world constraints that managers and engineers face, we developed a simple three-step procedure for creating rubrics that improve yes-or-no decisions. These rubrics can help judges decide whom to detain, tax auditors whom to scrutinize, and hiring managers whom to interview. Our approach offers practitioners the performance of state-of-the-art machine learning while stripping away needless complexity.

To see these rules in action, consider pretrial release decisions. When defendants first appear in court, judges must assess their likelihood of skipping subsequent court dates. Those deemed low-risk are released back into the community, while high-risk defendants are detained in jail; these decisions are thus consequential both for defendants and for the general public. To aid judges in making these decisions, we used our procedure to create the simple risk chart below. Each defendant’s flight risk is computed by summing scores corresponding to their age and number of court dates missed. A risk threshold is then applied to convert the score to a binary release-or-detain recommendation. For example, with a risk threshold of 10, a 35-year-old defendant who has missed one court date would score an eight (two for age plus six for missing one prior court date), and would be recommended for release.


Despite its simplicity, this rule significantly outperforms expert human decision makers. We analyzed over 100,000 judicial pretrial release decisions in one of the largest cities in the country. Following our rule would allow judges in this jurisdiction to detain half as many defendants without appreciably increasing the number who fail to appear at court. How is that possible? Unaided judicial decisions are only weakly related to a defendant’s objective level of flight risk. Further, judges apply idiosyncratic standards, with some releasing 90% of defendants and others releasing only 50%. As a result, many high-risk defendants are released and many low-risk defendants are detained. Following our rubric would ensure defendants are treated equally, with only the highest-risk defendants detained, simultaneously improving the efficiency and equity of decisions.

Decision rules of this sort are fast, in that decisions can be made quickly, without a computer; frugal, in that they require only limited information to reach a decision; and clear, in that they expose the grounds on which decisions are made. Rules satisfying these criteria have many benefits, both in the judicial context and beyond. For instance, easily memorized rules are likely to be adopted and used consistently. In medicine, frugal rules may reduce tests required, which can save time, money, and, in the case of triage situations, lives. And the clarity of simple rules engenders trust by revealing how decisions are made and indicating where they can be improved. Clarity can even become a legal requirement when society demands fairness and transparency.

Simple rules certainly have their advantages, but one might reasonably wonder whether favoring simplicity means sacrificing performance. In many cases the answer, surprisingly, is no. We compared our simple rules to complex machine learning algorithms. In the case of judicial decisions, the risk chart above performed nearly identically to the best statistical risk assessment techniques. Replicating our analysis in 22 varied domains, we found that this phenomenon holds: Simple, transparent decision rules often perform on par with complex, opaque machine learning methods.
Related Video
To create these simple rules, we used a three-step strategy, detailed here, that we call select-regress-round. Here’s how it works.
Select a few leading indicators of the outcome in question — for example, using a defendant’s age and number of court dates missed to assess flight risk. We find that having two to five indicators works well. The two factors we used for pretrial decisions are well-known indicators of flight risk; without such domain knowledge, one can create the list of factors using standard statistical methods (e.g., stepwise feature selection).

Using historical data, regress the outcome (skipping court) on the selected predictors (age and number of court dates missed). This step can be carried out in one line of code with modern statistical software.

The output of the above step is a model that assigns complicated numerical weights to each factor. Such weights are overly precise for many decision-making applications, and so we round the weights to produce integer scores.

Our select-regress-round strategy yields decision rules that are simple. Equally important, the method for constructing the rules is itself simple. The three-step recipe can be followed by an analyst with limited training in statistics, using freely available software.

Statistical decision rules work best when objectives are clearly defined and when data is available on past outcomes and their leading indicators. When these criteria are satisfied, statistically informed decisions often outperform the experience and intuition of experts. Simple rules, and our simple strategy for creating them, bring the power of machine learning to the masses.
---------------------------------------------------------------------
Authors:
Jongbin Jung is a PhD candidate at Stanford University in the Department of Management Science & Engineering.
Connor Concannon is Deputy Director of Analytics at the New York County District Attorney’s Office and a PhD student at John Jay College of Criminal Justice.
Ravi Shroff is a Research Scientist in the Center for Urban Science and Progress at New York University.
Sharad Goel is an Assistant Professor at Stanford University in the Department of Management Science & Engineering.
Daniel G. Goldstein is a Principal Researcher at Microsoft Research.

Comments

Readers Choice

Lead Your Team Into a Post-Pandemic World

During the Covid-19 crisis, I’ve spoken with many CEOs who have shared that a key priority for them, naturally, has been the safety and well-being of their employees. And there are many examples of inspiring actions taken by CEOs and companies in support of their employees. But as we’ve come to recognize that this crisis will last more than a few short weeks, companies are now defining their approach for the long haul. I’ve seen two crucial ideas take hold with corporate leaders. One: Given the magnitude of the shock and the challenges that this crisis represents, companies must consider the full breadth of their employees’ needs as people. Safety is essential, of course, but it’s also important to address higher-level needs such as the want for truth, stability, authentic connections, self-esteem, growth, and meaning in the context of the crisis. Two: Many CEOs have begun thinking about this crisis in three phases. They may assign different names or specific lengths to t

4 Ways Google Search Can Help You Achieve Your Marketing Goals

Google Ads Google Ad extensions are a great way to add key descriptive text without taking up space in your actual ad and improve your quality score for even better results. It’s a win-win right? Google Maps Is your business discoverable on Google Maps? For small businesses, adding detailed information and the use of strategic keywords can be helpful for a better location optimization. Google Ranks SEO is vital for moving up in Google rankings. To climb up the ladder, incorporate top keywords in page titles, meta tags and focus on consistently delivering relevant content. Backlinking If SEO is the the only strategy you have, it is the right time to change that. Start adding backlinks to your content. Quality backlinks can further increase your brand authority. 

Twenty Smart Business Buzzwords

Some words may grate on your nerves, but business leaders are still using "disrupt," "synergy" and "ideate." You should too. Spend any amount of time in a corporate environment and you'll likely notice there are some words that seem to come up on a daily basis. Certain verbiage becomes part of the  corporate culture  and soon, you may feel as if you need to use it to fit in. While they can change from one day to the next, most corporate buzzwords have a positive meaning. They're used to boost morale and motivate everyone involved in the conversation. Here are 20 of the top business buzzwords that you should make an effort to work into your vocabulary. 1. Impact Impact is a powerful word that has become a favorite of business professionals.  Grammarians argue  that the word is being used improperly, urging you to use "affect" instead, but businesses love it. 2. Corporate Synergy Half of the people who use this term likely