NILG.AI

When it comes to AI models, we often expect them always to provide an answer. But what if we could trust them more when they choose to remain silent? This concept, known as the ‘Reject option‘, allows AI models to abstain from answering when they are not confident, opening up many applications in your business and reducing risk.

Why Should AI Models Abstain?

Just like human experts, AI models can gain our trust when they know when to say, “I don’t know“. This capacity to abstain from answering, known as the ‘reject option’, can be a game-changer for many applications.

Consider a model that diagnoses whether a patient has cancer. Would you trust a model that always gives a definitive ‘yes‘ or ‘no‘, or one that sometimes says, “I don’t know, let’s get a second opinion”? The same applies to a model monitoring a manufacturing pipeline for quality assurance. A model that can say, “I don’t know, let’s get a technician to check this” can be more reliable than one that always guesses.

How Can You Use the Reject Option in Business?

There are two main strategies for using the reject option in business: filtering and incremental deployment.

Filtering

In the filtering strategy, the AI model automates most of the cases and asks for help with the rest. This help could come from a second AI model or a human expert. This strategy essentially adds an additional stage to your workflow, a fallback option for when the AI model cannot provide an answer.

Incremental Deployment

In the incremental deployment strategy, you start by only covering the top cases where the model is most confident. As you gain trust in the model, you start giving it more and more cases, even if the AI is not that confident about them. This strategy allows for a safer deployment and is particularly useful for initial pilots.

How to Compute the Expected Impact?

To compute the expected impact of using the reject option, you need to understand the trade-off between the percentage of times the model abstains and its predictive performance.

You start by computing how much benefit you would get from the model predicting 100% of cases and how much you would lose from the errors. Then, for any given percentage of abstentions, you calculate how much you would gain from the cases the AI automates and how much you would spend on the cases it doesn’t.

Remember, there is always a fixed cost for both the AI (infrastructure) and the human component (salaries), and an elastic cost for each case, depending on how much the AI abstains or predicts.

How to Get Models That Can Abstain?

The easiest way to have models that can abstain is by looking at the tails of your predictions and only predicting for those cases. This approach involves setting two thresholds, one for the lower bound and one for the upper bound, and choosing these thresholds to maximize your return.

However, this approach may not be enough. A more advanced approach involves allowing your model to predict which cases it wants to answer and what the answer should be for those cases. This can be achieved by having two outputs from your model: one indicating whether the model will abstain or not, and the other providing the answer if it’s not abstaining.

Conclusion

Allowing your AI models to abstain when they are not confident can open up many applications in your business and reduce risk. It’s a strategy worth considering as you plan your AI deployment. If you want to embrace smarter AI decisions, book a strategic meeting with our experts and transform risk into reliability today!