On Rashomon sets, the mathematics of simplicity, and why we don’t need black boxes: an interview with Cynthia Rudin

Welcome back to AI Pioneers – in-depth conversations with those shaping the field. This time, we speak with Cynthia Rudin, a trailblazer in the field of interpretable machine learning. Winner of the 2022 Squirrel AI Award for Artificial Intelligence for the Benefit of Humanity, Cynthia’s algorithms are already predicting seizures, aiding crime detection, and powering biological research. We discuss black boxes, Rashomon sets, and what’s next for her lab – from cancer detection to interpretable AI-generated music.

Can you tell me a bit about your background – what drew you into the field of interpretable machine learning?

I was working with Con Edison on maintaining the NYC power grid with machine learning. It was the first ever large-scale attempt to maintain a power grid with machine learning. The data was very messy – trouble tickets that were hardly understandable, cable installation records dating back to the 1880s, and tons of other tables that had no documentation. I initially tried to use a black box machine learning method to identify weaknesses in the power grid and it was a disaster. I was making suggestions of what to repair that made no sense.

But when I stopped trying to get black boxes to work and switched to very simple models, I was able to show the power engineers the interpretable models and they helped us troubleshoot the models and the data. Then our accuracy got much better than the black boxes ever hoped to get. It was eye-opening.

Can you tell me more about your theory about simpler models emerging from noise in the data?

There’s always been some kind of mystique around black boxes, where people think they must always be more accurate because they are more complicated. But for a lot of high-stakes problems like loan decisions or criminal risk predictions or a lot of different healthcare problems, it’s just not true. The data are noisy enough that statistically, they can’t support that level of complexity. I had a lot of trouble convincing people of this, and it was frustrating because it is just so obvious empirically, even from the benchmark datasets that are commonly used in machine learning experiments.

So I decided it was worthwhile to try to prove it since it’s harder to argue with a mathematical proof as opposed to an experimental result. And now my collaborators and I have several results showing why, under very general conditions when there is noise in the data, simple models will perform as well as more complex models. The noise creates uncertainty, and because of that uncertainty, statistically it is more difficult to tell apart simple models from complex models, which is why simple models are able to perform as well as the more complex models.

What are Rashomon Sets, and what implications do they have for the field of machine learning?

The Rashomon Effect is the phenomenon that many datasets admit many equally good models. (This phenomenon was named by the famous statistician Leo Breiman, which was named after the Japanese movie “Rashomon”). When you fit a machine learning model to a dataset, it’s likely not the only model you could have gotten. There are probably many other models that are completely different from each other that fit the data equally well – we call that set of good models the “Rashomon set.” The Rashomon Effect has been observed by statisticians since at least as far back as the 1980’s but only now have we begun to sort of unleash its power. Because if you truly understand the Rashomon Effect, it changes everything about the way you think about modeling with data.

When you think of the Rashomon Effect, you want to think of a large sea of good models for a given dataset. When the sea is larger, there can be more interesting fish in it. For instance, the Rashomon Effect gives us the existence of simpler-yet-accurate models, since if there are many good models, then often some of those models are simpler. The Rashomon Effect also allows us to incorporate other types of constraints into the models, like fairness constraints, since within the sea of good models, there are often models that obey the constraints. The Rashomon Effect also allows us a better understanding of which variables are important. Maybe you choose one good model from the Rashomon set and you see that a variable is important. It doesn’t mean that variable is important overall, because there could be another good model for which that variable is unimportant. So if you look at the whole Rashomon set, you can find out which variables are always important to all the good models, which variables are never important, and which variables are sometimes important but not always important.

You’ve already seen some of your algorithms utilised in a wide range of contexts: from bioinformatics and ecology research to critical care and crime detection. What have these very different – and high-stakes – domains taught you about designing useful AI?

I really don’t like to deploy anything I don’t understand if I don’t have to. What scares me is that not everyone feels the same way! There are a lot of people who are ok with deploying a black box even in cases where I know there is enough noise in the data that I’m quite certain a much simpler model could suffice with the same accuracy.

You can do a lot of troubleshooting with simpler models. Domain experts can really critique a model when they understand it – and that’s what creates the dialogue that allows your team to build a better model.

Do you think we will still have black-box models in the future?

Sure. There are many domains where it doesn’t matter if your model isn’t perfect. Machine learning grew up around domains like advertising and labeling images of dogs on Facebook. There are a lot of cases where we just don’t care too much if things go wrong, and black box models are generally fine for those cases.

There are also cases where black box models are all we need because the output can be corrected or verified afterwards by a human. For instance, outlining the contours of organs in medical images can be done by a black box and fixed by a human afterwards.

There will always be people who want to sell models, and interpretable models are harder to sell, so they will use the black boxes to obfuscate proprietary technology.

Also we currently have no idea how to make large language models interpretable, because we don’t know what interpretability means for them, so that’s an important direction for future work.

What future work do you have planned?

Some of my students are working on calculating Rashomon sets in computationally feasible ways, since the Rashomon sets are so useful in creating a dialogue between the humans and the machine learning algorithms.

Another set of students is working in dimension reduction for data visualization, where we try to project high dimensional data to 2D while preserving its structure. We already have upwards of 25K downloads per month of our PaCMAP algorithm, including scientists from all different domains.

I have students creating mathematical proofs of the existence of simpler models under different noise models – I love this line of work.

Several of my students work on variable importance with the Rashomon set. It is quite powerful since it is model-free.

I have a long-standing collaboration of students and radiologists that is working on computer-aided mammography. We found a new imaging marker (a type of asymmetry between the two breasts) that can predict cancer years in advance. It was one of the most popular papers in Radiology in 2024.

I have other students working in various domains, for instance, I have students creating a new interpretability framework for computer vision, I have students working on materials science, protein binding, neurology, criminal justice, and heart monitoring with wearable devices. Finally, I have a group working on AI-generated music that is interpretable, and thus can interface naturally with composers.

About Cynthia Rudin

Cynthia Rudin is the Gilbert, Louis, and Edward Lehrman Distinguished Professor of Computer Science at Duke University. She directs the Interpretable Machine Learning Lab, and her goal is to design predictive models that people can understand. Her lab applies machine learning in many areas, such as healthcare, criminal justice, and energy reliability. She holds degrees from the University at Buffalo and Princeton. She is the recipient of the 2022 Squirrel AI Award for Artificial Intelligence for the Benefit of Humanity from the Association for the Advancement of Artificial Intelligence (the “Nobel Prize of AI”), as well as the INFORMS Society of Data Mining Prize in 2024. She received a 2022 Guggenheim fellowship, and is a fellow of the American Statistical Association, the Institute of Mathematical Statistics, and the Association for the Advancement of Artificial Intelligence.

Ella Scallan is Assistant Editor for AIhub

Ella Scallan is Assistant Editor for AIhub